{"id":4574,"date":"2020-03-22T09:00:10","date_gmt":"2020-03-22T16:00:10","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=4574"},"modified":"2020-03-22T11:21:19","modified_gmt":"2020-03-22T18:21:19","slug":"my-power9-cpu-core-layout","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2020\/03\/22\/my-power9-cpu-core-layout\/","title":{"rendered":"My POWER9 CPU Core Layout"},"content":{"rendered":"\n

So, following on from my post on Sensors on the Blackbird (and thus Power9), I mentioned that when you look at the temperature sensors for each CPU core in my 8-core POWER9 chip, they’re not linear numbers. Let’s look at what that means….<\/p>\n\n\n\n

stewart@blackbird9$ sudo ipmitool sensor | grep core\n p0_core0_temp            | na                                                                                                               \n p0_core1_temp            | na                                                                                                               \n p0_core2_temp            | na                                                                                                               \n p0_core3_temp            | 38.000                                                                                                           \n p0_core4_temp            | na          \n p0_core5_temp            | 38.000      \n p0_core6_temp            | na          \n p0_core7_temp            | 38.000      \n p0_core8_temp            | na          \n p0_core9_temp            | na          \n p0_core10_temp           | na          \n p0_core11_temp           | 37.000      \n p0_core12_temp           | na          \n p0_core13_temp           | na          \n p0_core14_temp           | na          \n p0_core15_temp           | 37.000      \n p0_core16_temp           | na          \n p0_core17_temp           | 37.000      \n p0_core18_temp           | na          \n p0_core19_temp           | 39.000      \n p0_core20_temp           | na          \n p0_core21_temp           | 39.000      \n p0_core22_temp           | na          \n p0_core23_temp           | na        <\/pre>\n\n\n\n
You can see I have eight CPU cores in my Blackbird system. The reason the 8 CPU cores are core 3, 5, 7, 11, 15, 17, 19, and 21 rather than 0-8 or something is that these represent the core numbers on the physical die, and the die is a 24 core die. When you’re making a chip as big and as complex as modern high performance CPUs, not all of the chips coming out of your fab are going to be perfect, so this is how you get different models in the line with only one production line.<\/p>\n\n\n\n
Weirdly, the output from the hwmon sensors and why there’s a “core 24” and a “core 28”. That’s just… wrong. What it is, however, is right if you think of 8*4=32. This is a product of Linux thinking that Thread=Core in some ways. So, yeah, this numbering is the first thread of each logical core.<\/p>\n\n\n\n
[stewart@blackbird9 ~]$ sensors|grep -i core\n Chip 0 Core 0:            +39.0\u00c2\u00b0C  (lowest = +25.0\u00c2\u00b0C, highest = +71.0\u00c2\u00b0C)\n Chip 0 Core 4:            +39.0\u00c2\u00b0C  (lowest = +26.0\u00c2\u00b0C, highest = +66.0\u00c2\u00b0C)\n Chip 0 Core 8:            +39.0\u00c2\u00b0C  (lowest = +27.0\u00c2\u00b0C, highest = +67.0\u00c2\u00b0C)\n Chip 0 Core 12:           +39.0\u00c2\u00b0C  (lowest = +26.0\u00c2\u00b0C, highest = +67.0\u00c2\u00b0C)\n Chip 0 Core 16:           +39.0\u00c2\u00b0C  (lowest = +25.0\u00c2\u00b0C, highest = +67.0\u00c2\u00b0C)\n Chip 0 Core 20:           +39.0\u00c2\u00b0C  (lowest = +26.0\u00c2\u00b0C, highest = +69.0\u00c2\u00b0C)\n Chip 0 Core 24:           +39.0\u00c2\u00b0C  (lowest = +27.0\u00c2\u00b0C, highest = +67.0\u00c2\u00b0C)\n Chip 0 Core 28:           +39.0\u00c2\u00b0C  (lowest = +27.0\u00c2\u00b0C, highest = +64.0\u00c2\u00b0C)<\/pre>\n\n\n\n
But let’s ignore that, go from the IPMI sensors (which also match what the OCC shows with “occtoolp9 -LS<\/code>” (see below).<\/p>\n\n\n\n
$ .\/occtoolp9 -SL\nSensor Details: (found 86 sensors, details only for Status of 0x00)                                           \n     GUID Name             Sample     Min    Max U    Stat   Accum     UpdFreq   ScaleFactr   Loc   Type \n....\n   0x00ED TEMPC03\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     47      29     47 C    0x00 0x00037CF2 0x00007D00 0x00000100 0x0040 0x0008\n   0x00EF TEMPC05\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     37      26     39 C    0x00 0x00014E53 0x00007D00 0x00000100 0x0040 0x0008\n   0x00F1 TEMPC07\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     46      28     46 C    0x00 0x0001A777 0x00007D00 0x00000100 0x0040 0x0008\n   0x00F5 TEMPC11\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     44      27     45 C    0x00 0x00018402 0x00007D00 0x00000100 0x0040 0x0008\n   0x00F9 TEMPC15\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     36      25     43 C    0x00 0x000183BC 0x00007D00 0x00000100 0x0040 0x0008\n   0x00FB TEMPC17\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     38      28     41 C    0x00 0x00015474 0x00007D00 0x00000100 0x0040 0x0008\n   0x00FD TEMPC19\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     43      27     44 C    0x00 0x00016589 0x00007D00 0x00000100 0x0040 0x0008\n   0x00FF TEMPC21\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6\u00e2\u20ac\u00a6     36      30     40 C    0x00 0x00015CA9 0x00007D00 0x00000100 0x0040 0x0008<\/pre>\n\n\n\n
So what does that mean for physical layout? Well, like all modern high performance chips, the POWER9 is modular, with a bunch of logic being replicated all over the die. The most notable duplicated parts are the core (replicated 24 times!) and cache structures. Less so are memory controllers and PCI hardware.<\/p>\n\n\n\n
\"\"
P9 chip layout from page 31 of the POWER9 Register Specification<\/a><\/figcaption><\/figure>\n\n\n\n
See that each core (e.g. EC00 and EC01) is paired with the cache block (EC00 and EC01 with EP00). That’s two POWER9 cores with one 512KB L2 cache and one 10MB L3 cache.<\/p>\n\n\n\n
You can see the cache layout (including L1 Instruction and Data caches) by looking in sysfs<\/code>:<\/p>\n\n\n\n
$ for i in \/sys\/devices\/system\/cpu\/cpu0\/cache\/index*\/; \\\n  do echo -n $(cat $i\/level) $(cat $i\/size) $(cat $i\/type); \\\n  echo; done\n 1 32K Data\n 1 32K Instruction\n 2 512K Unified\n 3 10240K Unified<\/pre>\n\n\n\n
So, what does the layout of my<\/strong> POWER9 chip look like? Well, thanks to the power of graphics software, we can cross some cores out and look at the topology:<\/p>\n\n\n\n
\"\"
My 8-core POWER9 CPU in my Raptor Blackbird<\/figcaption><\/figure>\n\n\n\n
If I run some memory bandwidth benchmarks, I can see that you can see the L3 cache capacity you’d assume from the above diagram: 80MB (10MB\/core). Let’s see:<\/p>\n\n\n\n
[stewart@blackbird9 lmbench3]$ for i in 5M 10M 20M 30M 40M 50M 60M 70M 80M 500M; \\\n  do echo -n \"$i   \"; \\\n  .\/bin\/bw_mem -N 100  $i rd; \\\ndone\n  5M    5.24 63971.98\n 10M   10.49 31940.14\n 20M   20.97 17620.16\n 30M   31.46 18540.64\n 40M   41.94 18831.06\n 50M   52.43 17372.03\n 60M   62.91 16072.18\n 70M   73.40 14873.42\n 80M   83.89 14150.82\n 500M 524.29 14421.35<\/pre>\n\n\n\n
If all the cores were packed together, I’d expect that cliff to be a lot sooner.<\/p>\n\n\n\n
So how does this compare to other machines I have around? Well, let’s look at my Ryzen 7. Specifically, a “AMD Ryzen 7 1700 Eight-Core Processor”. The cache layout is:<\/p>\n\n\n\n
$ for i in \/sys\/devices\/system\/cpu\/cpu0\/cache\/index*\/; \\\n  do echo -n $(cat $i\/level) $(cat $i\/size) $(cat $i\/type); \\\n  echo; \\\ndone\n 1 32K Data\n 1 64K Instruction\n 2 512K Unified\n 3 8192K Unified<\/pre>\n\n\n\n
And then the performance benchmark similar to the one I ran above on the POWER9 (lower numbers down low as 8MB is less than 10MB)<\/p>\n\n\n\n
$ for i in 4M 8M 16M 24M 32M 40M 48M 56M 64M 72M 80M 500M; \\\n  do echo -n \"$i   \"; .\/bin\/x86_64-linux-gnu\/bw_mem -N 10  $i rd;\\\ndone\n  4M    4.19 61111.04\n  8M    8.39 28596.55\n 16M   16.78 21415.12\n 24M   25.17 20153.57\n 32M   33.55 20448.20\n 40M   41.94 20940.11\n 48M   50.33 20281.39\n 56M   58.72 21600.24\n 64M   67.11 21284.13\n 72M   75.50 20596.18\n 80M   83.89 20802.40\n 500M 524.29 21489.27<\/pre>\n\n\n\n
And my laptop? It’s a four core part, specifically a “Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz<\/code>” with a cache layout like:<\/p>\n\n\n\n
$ for i in \/sys\/devices\/system\/cpu\/cpu0\/cache\/index*\/; \\\n   do echo -n $(cat $i\/level) $(cat $i\/size) $(cat $i\/type); \\\n     echo; \\\n   done\n   1 32K Data\n   1 32K Instruction\n   2 256K Unified\n   3 6144K Unified <\/pre>\n\n\n\n
$ for i in 3M 6M 12M 18M 24M 30M 36M 42M 500M; \\\n  do echo -n \"$i   \"; .\/bin\/x86_64-linux-gnu\/bw_mem -N 10  $i rd;\\\ndone\n  3M    3.15 48500.24\n  6M    6.29 27144.16\n 12M   12.58 18731.80\n 18M   18.87 17757.74\n 24M   25.17 17154.12\n 30M   31.46 17135.87\n 36M   37.75 16899.75\n 42M   44.04 16865.44\n 500M 524.29 16817.10<\/pre>\n\n\n\n
I’m not sure what performance conclusions we can realistically draw from these curves, apart from “keeping workload to L3 cache is cool”, and “different chips have different cache hardware”, and “I should probably go and read and remember more about the microarchitectural characteristics of the cache hardware in Ryzen 7 hardware and 10th gen Intel Core hardware”.<\/p>\n","protected":false},"excerpt":{"rendered":"
So, following on from my post on Sensors on the Blackbird (and thus Power9), I mentioned that when you look at the temperature sensors for each CPU core in my 8-core POWER9 chip, they’re not linear numbers. Let’s look at … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[740,748,587,579,657],"class_list":["post-4574","post","type-post","status-publish","format-standard","hentry","category-general","tag-blackbird","tag-cache","tag-openpower","tag-performance","tag-power9"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-1bM","jetpack-related-posts":[{"id":4571,"url":"https:\/\/www.flamingspork.com\/blog\/2020\/03\/21\/occ-and-sensors-on-the-raptor-blackbird-and-other-power9-systems\/","url_meta":{"origin":4574,"position":0},"title":"OCC and Sensors on the Raptor Blackbird (and other POWER9 systems)","author":"Stewart Smith","date":"2020-03-21","format":false,"excerpt":"This post we're going to look at three different ways to look at various sensors in the Raptor Blackbird system. The Blackbird is a single socket uATX board for the POWER9 processor. One advantage of the system is completely open source firmware, so you can (like I have): build your\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2020\/03\/Screenshot-from-2020-03-21-12-52-30.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2020\/03\/Screenshot-from-2020-03-21-12-52-30.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2020\/03\/Screenshot-from-2020-03-21-12-52-30.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":4495,"url":"https:\/\/www.flamingspork.com\/blog\/2019\/12\/10\/blackbird-singing-in-the-dead-of-night\/","url_meta":{"origin":4574,"position":1},"title":"Blackbird (singing in the dead of night..)","author":"Stewart Smith","date":"2019-12-10","format":false,"excerpt":"Way back when Raptor Computer Systems was doing pre-orders for the microATX Blackboard POWER9 system, I put in a pre-order. Since then, I've had a few life changes (such as moving to the US and starting to work for Amazon rather than IBM), but I've finally gone and done (most\u2026","rel":"","context":"In "cool gadgets"","block_context":{"text":"cool gadgets","link":"https:\/\/www.flamingspork.com\/blog\/category\/cool-gadgets\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/IMG_4007-scaled.jpg?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/IMG_4007-scaled.jpg?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/IMG_4007-scaled.jpg?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/IMG_4007-scaled.jpg?resize=700%2C400 2x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/IMG_4007-scaled.jpg?resize=1050%2C600 3x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/IMG_4007-scaled.jpg?resize=1400%2C800 4x"},"classes":[]},{"id":4532,"url":"https:\/\/www.flamingspork.com\/blog\/2019\/12\/31\/speeding-up-blackbird-boot-the-sbe\/","url_meta":{"origin":4574,"position":2},"title":"Speeding up Blackbird boot: the SBE","author":"Stewart Smith","date":"2019-12-31","format":false,"excerpt":"The Self Boot Engine (SBE) is a small embedded PPE42 core inside the POWER9 CPU which has the unenvious job of getting a single POWER9 core ready enough to start executing instructions out of L3 cache, and poking some instructions into said cache for the core to start executing. It's\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4055,"url":"https:\/\/www.flamingspork.com\/blog\/2016\/05\/10\/first-power9-bits-merged-into-skiboot-master\/","url_meta":{"origin":4574,"position":3},"title":"First POWER9 bits merged into skiboot master","author":"Stewart Smith","date":"2016-05-10","format":false,"excerpt":"I just merged in some base POWER9 support patches into skiboot. While this is in no way near complete or really enough to be interesting to anyone that isn't heavily involved in POWER9 development, it's nice to take upstream first and open source first so seriously that this level of\u2026","rel":"","context":"In "IBM"","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4072,"url":"https:\/\/www.flamingspork.com\/blog\/2016\/06\/20\/building-opal-firmware-for-power9\/","url_meta":{"origin":4574,"position":4},"title":"Building OPAL firmware for POWER9","author":"Stewart Smith","date":"2016-06-20","format":false,"excerpt":"Recently, we merged into the op-build project (the build scripts for OpenPOWER Firmware) a defconfig for building OPAL for (certain) POWER9 simulators. I won't bother linking over to articles on the POWER9 chip or schedule (there's search engines for that), but with this commit - if you happen to be\u2026","rel":"","context":"In "IBM"","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4508,"url":"https:\/\/www.flamingspork.com\/blog\/2019\/12\/10\/looking-at-the-state-of-blackbird-firmware\/","url_meta":{"origin":4574,"position":5},"title":"Looking at the state of Blackbird firmware","author":"Stewart Smith","date":"2019-12-10","format":false,"excerpt":"Having been somewhat involved in OpenPOWER firmware, I have a bunch of experience and opinions on maintaining firmware trees for products, what working with upstream looks like and all that. So, with my new Blackbird system I decided to take a bit of a look as to what the firmware\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4574","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=4574"}],"version-history":[{"count":3,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4574\/revisions"}],"predecessor-version":[{"id":4583,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4574\/revisions\/4583"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=4574"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=4574"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=4574"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}