[{"content":"If you\u0026rsquo;re running inference, training, AI experiments or a lab locally on Apple silicon, you\u0026rsquo;ll likely want to be able to monitor your workloads.\nIf you\u0026rsquo;re running your workloads on CPUs, you have the normal *top family of tools available like top, htop and btop. But for GPUs, you\u0026rsquo;ll need to reach for a different set of tools. The two that I\u0026rsquo;ve been testing are:\nPowerMetrics macmon PowerMetrics GPU Metrics The built-in tool on macOS is called PowerMetrics. This works, but there are a few problems. First off, you have to run it with sudo, which is pretty tedious. The other problem is that Apple hasn\u0026rsquo;t documented it very well, so I\u0026rsquo;ve had to make some assumptions in interpreting the data.\nAn example below:\nsudo powermetrics --samplers gpu_power This should give you output similar to this:\n*** Sampled system activity (Sat Jun 27 20:18:22 2026 +0100) (5009.31ms elapsed) ***= **** GPU usage **** GPU HW active frequency: 392 MHz GPU HW active residency: 38.91% (389 MHz: 39% 486 MHz: .09% 648 MHz: .08% 778 MHz: .21% 972 MHz: .01% 1296 MHz: 0%) GPU SW requested state: (P1 : 99% P2 : .24% P3 : .35% P4 : .46% P5 : .03% P6 : 0%) GPU SW state: (SW_P1 : 0% SW_P2 : 0% SW_P3 : 0% SW_P4 : 0% SW_P5 : 0% SW_P6 : 0%) GPU idle residency: 61.09% GPU Power: 149 mW This can be a little confusing to decipher:\nGPU HW active frequency GPU HW active frequency: 392 MHz: the currently active hardware (HW) frequency of the GPU. Apple silicon operates at a number of different discrete frequencies, which are dynamically selected based on active workloads, power consumption, thermals etc. These frequencies are approximately [0, 389, 486, 648, 778, 972, 1296] MHz (which you will see listed in the next line item). Note that 392 MHz isn\u0026rsquo;t one of the discrete values; it\u0026rsquo;s computed as a weighted average over the sampling period.\nGPU HW active residency GPU HW active residency: 38.91% (389 MHz: 39% 486 MHz: .09% 648 MHz: .08% 778 MHz: .21% 972 MHz: .01% 1296 MHz: 0%): here you can see a breakdown of active residency by discrete frequency. Again, the numbers look confusing, and I haven\u0026rsquo;t quite worked out how they are derived. Adding the discrete percentages gives a total of 39.39%, not 38.91%. This could be due to various factors such as non-atomic sampling, different underlying hardware counters being used etc, but the logic isn\u0026rsquo;t documented anywhere I have been able to find.\nGPU SW requested state GPU SW requested state: (P1 : 99% P2 : .24% P3 : .35% P4 : .46% P5 : .03% P6 : 0%): here you will see what the software (SW) is requesting, meaning the Metal driver and macOS power management deciding what they want. When a workload is submitted, the driver looks at the demand and requests a P-state. This is a policy decision, something like \u0026ldquo;we think we need P4 right now\u0026rdquo;. This ranges from P1 (lowest demand) to P6 (highest demand).\nGPU SW state GPU SW state: (SW_P1 : 0% SW_P2 : 0% SW_P3 : 0% SW_P4 : 0% SW_P5 : 0% SW_P6 : 0%): this is the actual time spent in each software state. Everything shows 0%, which is unexpected given the SW requested state above. This is likely a rounding or counter sampling artefact.\nGPU idle residency GPU idle residency: 61.09%: fairly self-explanatory: the percentage of time over the sample period that the GPU has been idle. This at least tallies to 100% against the GPU HW active residency.\nGPU Power GPU Power: 149 mW: again, fairly self-explanatory: the power consumption of the GPU over the sampling period in mW.\nAll useful stats, despite some skewed numbers. Here we can compare things like software demand vs hardware residency, performance states and resource demand, and a comparison between the two. However, this snapshot isn\u0026rsquo;t overly useful by itself. We want to hook into something more sophisticated and track these metrics over time during our training or inference runs. This is where macmon comes in.\nmacmon macmon is an open source tool from vladkens available on GitHub here: https://github.com/vladkens/macmon.\nYou can install it via brew:\nbrew install macmon And you can run a nice little TUI with:\nmacmon This gives some basic graphing of your vitals:\nE-CPU USAGE% @ FREQ MHz P-CPU USAGE% @ FREQ MHz RAM USAGE / CAPACITY GB (USAGE%) - SWAP USAGE / CAPACITY G GPU USAGE% @ FREQ MHz Power: PWR W (avg PWR W, max PWR W) - Total PWR W (TOTAL_PWR_MIN, TOTAL_PWR_MAX) CPU CPU_PWR W (CPU_PWR_MIN, CPU_PWR_MAX) ─ TEMP °C GPU GPU_PWR W (GPU_PWR_MIN, GPU_PWR_MAX) ─ TEMP °C ANE ANE_PWR W (ANE_PWR_MIN, ANE_PWR_MAX) Very cool, super useful for ad hoc monitoring. But even better is that if you run:\nmacmon serve You get two endpoints (served by default on port 9090, though note that Prometheus also uses this port by default, so check for conflicts if you\u0026rsquo;re running both):\n/json: JSON output of the metrics /metrics: Prometheus-compatible output for scraping The JSON output shows more than just the GPU stats we looked at earlier from PowerMetrics, although you can easily get the full set and more from PowerMetrics with sudo powermetrics. There are also many different filtering flags and options you can get with powermetrics --help. It\u0026rsquo;s nice that you can get clean JSON output of the main metrics from macmon with:\ncurl -s http://localhost:9090/json Which shows something like:\n{ \u0026#34;all_power\u0026#34;: 0.494245052337647, \u0026#34;ane_power\u0026#34;: 0, \u0026#34;cpu_power\u0026#34;: 0.339286744594574, \u0026#34;cpu_usage_pct\u0026#34;: 0.0710111409425736, \u0026#34;ecpu_usage\u0026#34;: [1512, 0.255764722824097], \u0026#34;gpu_power\u0026#34;: 0.154958307743073, \u0026#34;gpu_ram_power\u0026#34;: 0.0064940438605845, \u0026#34;gpu_usage\u0026#34;: [388, 0.132904022932053], \u0026#34;memory\u0026#34;: { \u0026#34;ram_total\u0026#34;: 68719476736, \u0026#34;ram_usage\u0026#34;: 33498726400, \u0026#34;swap_total\u0026#34;: 2147483648, \u0026#34;swap_usage\u0026#34;: 762118144 }, \u0026#34;pcpu_usage\u0026#34;: [1359, 0.0248227454721928], \u0026#34;ram_power\u0026#34;: 1.08980870246887, \u0026#34;soc\u0026#34;: { \u0026#34;chip_name\u0026#34;: \u0026#34;Apple M1 Max\u0026#34;, \u0026#34;ecpu_cores\u0026#34;: 2, \u0026#34;ecpu_freqs\u0026#34;: [600, 972, 1332, 1704, 2064], \u0026#34;ecpu_label\u0026#34;: \u0026#34;E\u0026#34;, \u0026#34;gpu_cores\u0026#34;: 32, \u0026#34;gpu_freqs\u0026#34;: [0, 388, 486, 648, 777, 972, 1296], \u0026#34;mac_model\u0026#34;: \u0026#34;MacBookPro18,2\u0026#34;, \u0026#34;memory_gb\u0026#34;: 64, \u0026#34;pcpu_cores\u0026#34;: 8, \u0026#34;pcpu_freqs\u0026#34;: [600, 828, 1056, 1296, 1524, 1752, 1980, 2208, 2448, 2676, 2904, 3036, 3132, 3168, 3228], \u0026#34;pcpu_label\u0026#34;: \u0026#34;P\u0026#34; }, \u0026#34;sys_power\u0026#34;: 9.14718246459961, \u0026#34;temp\u0026#34;: { \u0026#34;cpu_temp_avg\u0026#34;: 47.0178909301758, \u0026#34;gpu_temp_avg\u0026#34;: 45.5415573120117 }, \u0026#34;timestamp\u0026#34;: \u0026#34;2026-06-27T19:32:07.409680+00:00\u0026#34; } You can see that we have things like the GPU frequencies listed, the power, thermals etc. Very nice. You can pipe this to jq for a snapshot, read it from custom tooling etc.\nEven better though, is that you can set up a quick Prometheus and Grafana with docker compose and build out some dashboards. I\u0026rsquo;ve done a quick demo here: https://github.com/code-engine/apple-silicon-prometheus-grafana that you can use as a template to try it out.\nSummary I hope you\u0026rsquo;ll find this useful, whether experimenting with serving inference on your Mac Mini or doing AI / ML workloads on your MacBook.\n","permalink":"https://codeengine.io/posts/apple_silicon_gpu_monitoring/","summary":"\u003cp\u003eIf you\u0026rsquo;re running inference, training, AI experiments or a lab locally on Apple silicon, you\u0026rsquo;ll likely want to be able to monitor your workloads.\u003c/p\u003e\n\u003cp\u003eIf you\u0026rsquo;re running your workloads on CPUs, you have the normal \u003ccode\u003e*top\u003c/code\u003e family of tools available like \u003ccode\u003etop\u003c/code\u003e, \u003ccode\u003ehtop\u003c/code\u003e and \u003ccode\u003ebtop\u003c/code\u003e. But for GPUs, you\u0026rsquo;ll need to reach for a different set of tools. The two that I\u0026rsquo;ve been testing are:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePowerMetrics\u003c/li\u003e\n\u003cli\u003emacmon\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"powermetrics\"\u003ePowerMetrics\u003c/h2\u003e\n\u003ch3 id=\"gpu-metrics\"\u003eGPU Metrics\u003c/h3\u003e\n\u003cp\u003eThe built-in tool on macOS is called PowerMetrics. This works, but there are a few problems. First off, you have to run it with \u003ccode\u003esudo\u003c/code\u003e, which is pretty tedious. The other problem is that Apple hasn\u0026rsquo;t documented it very well, so I\u0026rsquo;ve had to make some assumptions in interpreting the data.\u003c/p\u003e","title":"Apple Silicon GPU Monitoring"},{"content":"With the proliferation and widespread adoption of LLMs, I\u0026rsquo;ve heard a lot of people talking about how they \u0026ldquo;lie\u0026rdquo; recently. I\u0026rsquo;ll admit, I\u0026rsquo;ve felt the urge to reach for that word myself, but it\u0026rsquo;s not quite right for what\u0026rsquo;s actually happening. LLMs don\u0026rsquo;t lie as such: the more accurate term is hallucination.\nLLMs are built on what is known as a corpus, which is all of the data used to train the model. If the information you\u0026rsquo;re looking for isn\u0026rsquo;t in the corpus, then the LLM can\u0026rsquo;t by itself give you the answer. What tends to happen in modern frontier systems is that the orchestration layer and the model will work together to try and fetch the missing information from the internet or any other storage the system is aware of. If that fails, the LLM will do one of two things:\nTell you it doesn\u0026rsquo;t know. More likely, hallucinate. Frontier models and their associated orchestration systems are getting better at the former, but there is still a long way to go. The key thing to understand is that an LLM cannot reason about the knowledge it has or does not have. It is a probabilistic system that takes natural language as an input and produces a result as an output. When you search for something on Google, if it\u0026rsquo;s not found, you can reason about that yourself, whereas an LLM will essentially answer, with confidence, most of the time.\nThis is critical to understand, especially if you are building applications and systems on top of an LLM, because if you\u0026rsquo;re not careful, the system can produce incorrect / erroneous results, with confidence.\nAs a concrete example of this, a while back, I wanted to find a specific (and apt) quote from Frank Herbert\u0026rsquo;s Dune. The quote is:\n“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” — Frank Herbert, Dune\nI was fairly certain the quote was in the book, I just couldn\u0026rsquo;t remember where and the exact wording. I thought to myself that this would be a trivial job for an LLM.\nI tried several of the frontier models, but not one of them could find it. More interestingly, they all had a go at it and gave very confident, but very wrong answers. One claimed it was in a section that it wasn\u0026rsquo;t, one said it wasn\u0026rsquo;t in Dune at all and that it was in fact in fan fiction designed to sound like Herbert, another said it was in an appendix, another said it was in some versions but not others, and finally another said it was in a different book series altogether.\nNone of this is true, the quote is in fact in the first Dune book. I found it in seconds on my ebook reader using a basic text search.\nSo what happened here? How could such sophisticated machinery be outdone by a basic keyword search?\nThe models didn\u0026rsquo;t lie, they did exactly what they were built to do, which is generate the most plausible answer they could from the data they had available.\nDune is not freely available on the open internet. The quote was not widely cited with the applicable chapter and verse online. I can assume with a high degree of certainty that it doesn\u0026rsquo;t exist in any of their training corpora in a way that would allow the model to find it. So each generated the most plausible sounding answer it could construct. And, because each model was trained and tuned on different data and parameters, each produced a slightly different answer.\nThis example isn\u0026rsquo;t unique either. Ask an LLM about recent sporting events, a breaking news story, or the internal data within your organisation and you will hit the same problem. The information will either postdate the training corpus, or doesn\u0026rsquo;t exist anywhere that the model can reach. Unfortunately, the failure mode is the same for each case, which is a confidently stated, fabricated answer.\nWhat this example hopefully illustrates is that there are broadly three classes of question you can ask an LLM:\nQuestions whose answers are well represented in training data Questions whose answers are not in the training data, but are retrievable from external sources Questions whose answers are not in the training data and are unretrievable The model is highly likely to respond to all three with the same tone of confidence. They have no reliable way of differentiating which class it is responding to and this is just a characteristic of how these systems work.\nIt\u0026rsquo;s worth noting that the better frontier models are actively trying to improve this. Some will have a confidence score associated with their responses and will sometimes be able to tell you when they are not sure, flag that the information might be outdated or decline the answer rather than confabulate. But it\u0026rsquo;s far from a solved problem, and not consistent enough to rely on. The habit of asking and understanding \u0026ldquo;could the model actually know this\u0026rdquo;, needs to be owned by you, not relied on the model to be self aware.\nAt the end of the day, LLMs are a tool. The users that will get the most out of them are the ones who understand their capabilities and limitations and design their workflows and applications accordingly.\nFor applications where accuracy matters, you cannot rely on the training corpus or web retrieval alone. You need to supply the model with your own data and information through RAG, knowledge graphs, fine tuning, MCP tooling etc, so that the model has access to the information it needs to service your query.\nI like to internalise this as the \u0026ldquo;Dune Test\u0026rdquo;. Before you trust an LLM to answer anything that matters, ask yourself this: \u0026ldquo;is the model likely to have access to this information?\u0026rdquo;. Is it on the open internet, well cited and widely referenced? Is it locked in a book or a private data estate?\nThis will hopefully help you when asking LLMs questions and understand the importance of data accessibility when designing LLM based solutions.\n","permalink":"https://codeengine.io/posts/the_dune_test/","summary":"\u003cp\u003eWith the proliferation and widespread adoption of LLMs, I\u0026rsquo;ve heard a lot of people talking about how they \u0026ldquo;lie\u0026rdquo; recently. I\u0026rsquo;ll admit, I\u0026rsquo;ve felt the urge to reach for that word myself, but it\u0026rsquo;s not quite right for what\u0026rsquo;s actually happening. LLMs don\u0026rsquo;t lie as such: the more accurate term is \u003cem\u003ehallucination\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eLLMs are built on what is known as a corpus, which is all of the data used to train the model. If the information you\u0026rsquo;re looking for isn\u0026rsquo;t in the corpus, then the LLM can\u0026rsquo;t by itself give you the answer. What tends to happen in modern frontier systems is that the orchestration layer and the model will work together to try and fetch the missing information from the internet or any other storage the system is aware of. If that fails, the LLM will do one of two things:\u003c/p\u003e","title":"The Dune Test"},{"content":"I have a lot of discussions about AI these days, in fact, it\u0026rsquo;s almost exclusively what I talk about in a work context. Some months ago, I found myself in a difficult position, in trying to wrangle with all of the different components that I was discussing. There is a rapidly growing and sprawling landscape to try and get your head around and I eventually had to sit down and try and model it in a simple, describable way.\nBelow I\u0026rsquo;ve shared my thinking in two sections: 5 layers and 5 concerns that I use to group these topics into a comprehensive mental model.\nThe 5 Layers The 5 core layers attempts to cover the complete landscape of AI today:\nLayer Question Examples Application What does the user see and interact with? Products, integrations, APIs, productivity tools, coding assistants, consumer applications Orchestration How is the model directed and coordinated? Agents, multi-agent workflows, prompt engineering, LangChain/LangGraph, model routing, hybrid local/remote workflows, multi-model pipelines Model What intelligence is being used and how is it configured? Model selection, training, fine-tuning, inference serving, self-hosted vs API, local vs remote Access How does the model reach the world beyond its training? MCP, RAG, knowledge graphs, context graphs, embeddings, tool calls, APIs, memory retrieval Data What information exists and where does it live? Relational databases, blob stores, vector stores, data lakes, lakehouses, time-series databases, websites, unstructured data The 5 Concerns The next five are related but separate concerns that a business or adopter of AI should be thinking about.\nConcern Question Examples Capability What can we do, and why does it matter? Industry capability frontier, organisational maturity, adoption readiness, use case selection, skills and talent, strategic intent Economics Is it worth doing, and can we afford it? Cost of inference, infrastructure spend, build vs buy decisions, ROI, investment prioritisation, pricing models Controls How do we protect and constrain it? Access control, data security, guardrails, misuse prevention, prompt injection, audit trails, cyber threats Operability How do we run and sustain it? Observability, monitoring, MLOps/LLMOps, scalability, reliability, incident response, testing and evaluation, performance Governance How do we keep ourselves accountable for it? Ethics, compliance, regulation, responsible AI policy, model risk, bias and fairness, explainability, data governance I use these when discussing AI with clients, from strategy to technical implementation. I\u0026rsquo;ve refined and tweaked this over the last few months. In upcoming posts, I\u0026rsquo;ll likely refer back to these lists quite a bit. I\u0026rsquo;ve found it a very useful tool, I hope you will too.\n","permalink":"https://codeengine.io/posts/5_layers_5_concerns_modern_ai/","summary":"\u003cp\u003eI have a lot of discussions about AI these days, in fact, it\u0026rsquo;s almost exclusively what I talk about in a work context. Some months ago, I found myself in a difficult position, in trying to wrangle with all of the different components that I was discussing. There is a rapidly growing and sprawling landscape to try and get your head around and I eventually had to sit down and try and model it in a simple, describable way.\u003c/p\u003e","title":"The 5 Layers and 5 Concerns of Modern AI"}]