Category Archives: Programming

Building R packages from source on MacOS

Posted on by

For some reason Apple decided to break building C/C++ on MacOS by moving the C and C++ stdlib headers into a rando location that most toolchains don’t understand. Amazing. I ran into this while trying to build the R ggplot2 package from source and got missing header errors.

When installing a package from source fails, you’ll see a message like this.
Warning in install.packages :
installation of package ‘Rcpp’ had non-zero exit status

If you look back in the build output, you’ll see errors about missing headers. Something like this:
In file included from api.cpp:26:
In file included from ../inst/include/Rcpp.h:27:
In file included from ../inst/include/RcppCommon.h:30:
In file included from ../inst/include/Rcpp/r/headers.h:66:
../inst/include/Rcpp/platform/compiler.h:100:10: fatal error: 'cmath' file not found

The problem is that Apple moved these files out of /usr/include and /usr/local/include into a magical SDK package that most things don’t know about. Luckily R provides a way to set up the build environment when building packages from source with a file called Makevars.

First you need to find the location of your headers. Assuming you already have the Xcode command line tools installed, run xcrun --show-sdk-path
> xcrun --show-sdk-path
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

In that package, you’ll find usr/include — but that’s not where the C++ headers are either. They are actually in usr/include/c++/v1.

Next, you need to create the Makevars file ~/.R/Makevars — you may need to create the .R folder first. (mkdir ~/.R)

Once you’ve created the Makevars file, add this build variable and save:
CPPFLAGS = -I /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 -I /opt/homebrew/include

Open an R command prompt and run tools::makevars_user() — you should see the path to your new Makevars file if you put it in the right place.
[1] "/Users/username/.R/Makevars"

You should be good to go.

UPDATE – I still ran into more problems while trying to install the devtools package and needed additional tweaks. For reference, I’m trying to follow the setup steps for the book “The Art of Machine Learning” by Maloff where the ultimate goal of all this mess is to get install_github("https://github.com/matloff/qeML") to install correctly.

  • brew install libgit2

My Makevars file now looks like this hack-a-palooza. The R source packages are pretty dirty and their build scripts seem to need a lot of help selecting C++ version especially.
INCLUDE_PATH = -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/opt/homebrew/include
CPPFLAGS = $(INCLUDE_PATH)
CFLAGS = $(INCLUDE_PATH)
CC = clang $(INCLUDE_PATH)
CXX = clang++ -std=c++17 $(INCLUDE_PATH)
CXX11 = clang++ -std=c++11 $(INCLUDE_PATH)
CXX14 = clang++ -std=c++14 $(INCLUDE_PATH)
CXX17 = clang++ -std=c++17 $(INCLUDE_PATH)
OBJCXX = clang++ -std=c++17 $(INCLUDE_PATH)

References:

Built-in Columns in JSON View Forms

Posted on by

SharePoint Online supports custom list view formatting with a funky little JSON DSL that is meant to allow customization without giving users full control over HTML or Javascript which is unsafe. This feature was added back in 2017.

The way it works, only displayed columns are allowed to be referenced in the custom JSON. Unfortunately, the feature owners never got around with supporting the “hidden” or “default” SharePoint columns like Created By or Last Modified. As a result, there is no way to show these columns at all in the “modern UI”. (As of 2024, this is still the case and it boggles the mind).

At work we have these “hack day”-style events that we call FHL (Fix. Hack. Learn.) where we are given some time to work on knocking out some of the little problems that get in the way of work, annoy us, QOL improvements, etc. This bug was bothering me because we are using some SharePoint lists internally to track stuff and we often want to see who created an item in the list, but there was no way to show it without some crazy hacks. (Dear feature owners: using a workflow to copy a default field into a totally redundant one on the same item is not my idea of a workaround.)

Long story short, I added hacky support for some of the default columns so they can be used in custom JSON views. The following columns should be able to be referenced now in SharePoint Online.

  • $Author — The “Created by” user. This a user object, e.g. [$Author.title] or [$Author.email]
  • $Editor — The “Modified by” user.
  • $Created — The date the item was created.
  • $Modified — The date the item was last modified.
  • $ID — The list item id.
  • $_UIVersionString — The “pretty” string for the current item version.

Here’s the most basic demonstration of adding a custom footer to the list view form for a document.

{
    "elmType": "div",
    "style": {
        "display": "block"
    },
    "children": [
        {
            "elmType": "div",
            "txtContent": "[$Author.title]"
        },
        {
            "elmType": "div",
            "txtContent": "[$Created]"
        },
        {
            "elmType": "div",
            "txtContent": "[$Editor.title]"
        },
        {
            "elmType": "div",
            "txtContent": "[$Modified]"
        },
        {
            "elmType": "div",
            "txtContent": "[$ID]"
        },
        {
            "elmType": "div",
            "txtContent": "[$_UIVersionString]"
        }
    ]
}

Here are a few posts from others online who have been struggling with this. Hopefully they run across this post and get unblocked.

ETW and Async in .NET

Posted on by

I ran into this problem at work and wanted to capture the findings for future me.

ETW uses the ActivityId field on the TEB to track an activity. This gets logged with the ETW events.

When adding async Tasks to the mix, Tasks have an abstraction on the execution context that is per Task. Tasks may run on multiple different threads and so it’s possible that a Task gets hooked up to a thread that already has an activityId set. This messes up logging.

AsyncLocal provides a way to “flow” an ambient value across multiple Tasks as part of a logical operation. This can be used to track activityId on Tasks, but the trick is getting it pushed down onto the native thread.

AsyncLocal has a constructor argument that is a callback that is executed when the value changes. Values are always refreshed from empty when a Task is assigned to a thread. This provides a place where the activityId can be flowed down to the executing thread so that ETW logging works.

See ActivityTracker.cs on ReferenceSource to see how this works in .NET Framework.

Dirty-PocketChip-wave M8

Posted on by

Before I decided to buy the M8, I tried it “headless” with a Teensy for a while to see if I could figure it out. It worked well enough, but the whole point of the M8 is that it’s portable and fun. I thought I’d try to get it running on my PocketChip. It worked and it wasn’t even that bad to use. If I really wanted to I could have stuffed the Teensy into the Chip’s case and had a pretty decent portable experience. I even made a song on it.

I wrote a short readme and have the source on my GitHub. There’s also a pre-built binary if you have a Teensy and a PocketChip and want to give it a try.

Setting a Person field in SharePoint

Posted on by

This shouldn’t be this hard. The SharePoint team really needs to work on the programming model. This web API is bad bad.

For “reasons” I found myself needing to update a SharePoint list item to set the value of a Person field. It turns out that this is insanely annoying because what the ListItem API expects is the resolved User Id (it’s some rando number that’s assigned to the user when they access the site for the first time.) I’m working in a PowerAutomate flow and unfortunately that’s not a value that comes along with a user’s object. That would be too helpful.

If you try to use the Claims string to set the PersonFieldStringId field in a MERGE, it just doesn’t work. You get some terrible error message like Bad Gateway or something equally useless. Also, if you try to set through the navigation property like PersonField: { "Claims": ... }, that will succeed, but not actually work.

After being very frustrated and increasingly annoyed, I finally found someone who said to use the validateUpdateListItem method on the item. It worked. Apparently, this method “Validates and sets the values of the specified collection of fields for the list item.”

In case you find yourself on this site, wondering how this can all be so unnecessarily complicated, here’s the request you want to make.

URI: https://tenant.sharepoint.com/sites/MySite/_api/web/lists/GetByTitle('My List Name')/items(1234)/validateUpdateListItem
Method: POST
Headers: Content-Type=application/json;odata=verbose
Body:
{
  "formValues": [
    {
      "FieldName": "Your_x0020_Field_x0020_Name",
      "FieldValue": "[{'Key':'i:0#.f|membership|user@domain.com'}]"
    }
  ]
}

PICO-8 Generic Dispatch with Parameters

Posted on by

PICO-8 has this nice helper function called foreach(table, func) which will call the function for each item. I like to use this in my _draw function to dispatch drawing to game objects. The trouble is that most of the time I want to call a method (table-bound function) instead of a function. Normally this required a helper function per-method to invoke.

Consider the following PICO-8 code (it’s just Lua with some sugar).

-- returns a table with table-bound method 'draw'
function dot(x, y)
 return {x=x, y=y,
  draw=function(self,col)
   local c=col or 7
  	pset(self.x,self.y,c)
  end
 }
end

-- returns a function with captured upvalues
function dot2(x, y)
 return function(col)
  local c=col or 7
  pset(x,y,c)
 end
end

function _init()
 drawable={}
 add(drawable, dot(64, 64))
 add(drawable, dot2(64, 68))
end

In this example, I’m showing two different ways to make “drawable” instances. dot returns its instance as a table with a method named draw. dot2 returns a function which draws itself when invoked. I’ll start with the dot2 case.

function invoke(o)
 o()
end

function _draw()
 cls()
 foreach(drawable, invoke)
end

The nice thing about the approach of making drawable a function closure is that you can make a generic invoke helper function which just calls instance. This means you only need a single helper function. The downside is that you can only return one callable behavior this way which isn’t ideal for more complex objects.

function call_draw(o)
 o:draw()
end

function _draw()
 cls()
 foreach(drawable, call_draw)
end

This approach is more flexible and allows for more methods to be defined, but now you need a call_foo helper for each different method to call. Another problem with both approaches is you can’t pass in arguments to the method calls.

It turns out that Lua gives us a ton of flexibility to build our generic dispatch. Let’s start with the problem of dispatching by name.

-- Call Method
function callm(method)
 return function(o)
  o[method](o)
 end
end

function _draw()
 cls()
 foreach(drawable, callm("draw"))
end

callm is a pretty simple helper. The core of it just does a table lookup by name and invokes the result. callm returns a function because foreach expects a function that takes the object being iterated. There are a few problems with this approach. First this crashes if there isn’t a draw method on the instance. Second, the dot.draw method takes an argument and there’s no way to pass one in.

-- Call Method
function callm(method, ...)
 local params={...}
 return function(o)
  if type(o)=="table" then
   local m=o[method]
   if type(m)=="function" then
    m(o,unpack(params))
   end
  end
 end
end

function _draw()
 cls()
 foreach(drawable, callm("draw", 10))
end

This version adds support for varargs that get forwarded to the method when the returned function is called. This allows parameters to be passed to the method. This also adds some type checking to ensure the instance passed in is a table and has a function named method. You can see the foreach call site looks pretty good.

Finally, if you want to continue to use callable objects instead of tables, we can take this same concept and make it work with those too.

-- Call function 'object'
function call(...)
 local params={...}
 return function(o)
  if type(o)=="function" then
   o(unpack(params))
  end
 end
end

function _draw()
 cls()
 foreach(drawable, call(10))
end

Final thoughts

This adds overhead to the calls, and it’s almost certainly more efficient to use for x in all(XS). That said, it’s a neat little piece of code and I think it’s pretty cool that it can be accomplished with Lua.

Try out the code.
function _init()
 game_items={}
 -- add items
end

function _update()
 -- call 'update' on each instance that implements it
 foreach(game_items, callm('update')
end

function _draw()
 -- call 'draw' on each instance that implements it
 foreach(game_items, callm('draw')
end

P5.js GLSL Shaders on Retina Display

Posted on by

I’ve been playing around with GLSL Shaders and was trying out a few P5.js examples on my Mac. I couldn’t figure out why my results weren’t looking like the examples.

The following is the “Hello, world” of shaders. It simply renders a gradient from black to red smoothly across the canvas.

let header =
  'precision highp float;';

// the vertex shader is called for each vertex
let vs =
  header +
  'attribute vec3 aPosition;' +
  'void main() {' +
  '  vec4 positionVec4 = vec4(aPosition, 1.0);' +
  '  positionVec4.xy = positionVec4.xy * 2.0 - 1.0;' + // Correct for GL offset bug.
  '  gl_Position = positionVec4;' +
  '}';

// the fragment shader is called for each pixel
let fs =
  header +
  'uniform vec2 u_resolution;' +
  'void main() {' +
  '  vec2 st = gl_FragCoord.xy/u_resolution.xy;' +
  '  gl_FragColor = vec4(st.x,st.y,0.0,1.0);' + 
  '}';

let myShade;
function setup() {
  createCanvas(100, 100, WEBGL); // (0,0) is center
  //pixelDensity(1);  // <-- Uncomment to fix scaling.

  // create and initialize the shader
  myShade = createShader(vs, fs);
  shader(myShade);
  fill(64);
  stroke(128); 
}

function draw() {
  background(192);
  myShade.setUniform("u_resolution", [width, height]);
  rect(-40, -40, 80, 80);
}

Unfortunately when I run this example on my Mac the gradient stops halfway across the canvas.

That’s not right. It’s not supposed to be completely red until the right edge.

The problem is that by default there’s pixel scaling because of the Retina display which causes the pixel math to be incorrect. The solution is to specify the pixelDensity in the setup function. If the pixel density is set to 1, the shader works as expected.

Set pixelDensity(1) in the script setup to fix the shader scaling problem.