Welcome one and all to this announcement of our latest Beeminder integration: Curlminder. This one’s for the nerds.
The idea is that you give us a URL and a regular expression, and we fetch the contents of the webpage, match it on the regex, and pull out a number.
In theory you can use this to beemind arbitrary numbers on arbitrary web pages! With some caveats:
- You have to know regular expresssions well enough to extract the number you care about from the raw html of the page
- The page has to be technically publicly accessible [1]
- Some pages seem to have perfectly mindable numbers on them but when you fetch the html, it turns out the number is added via Javascript after the page is loaded. Sad face!
(If this sad-face scenario happens to you, do let us know. It’s possible for us to work around this by rendering the whole page with an emulated browser on the server side. We’re just not sure yet if it’s worth it.)
Can I see an example?
Say you’re me (Bee) and you have a Codewars profile viewable at codewars.com/users/bsoule and you want to beemind your XP, or “honor” as they call it. (The Honor is listed under “Progress”.)
First you’ll enter the URL and we fetch the page and show you the raw HTML that we fetched, so you can double check that we are getting what you think we are getting.
Notice that the raw HTML can be super verbose and probably overwhelming, like this.
We recommend using ctrl + f to search for whatever keywords you’re looking for, to make sure they’re there.
Probably copying that snippet and a bit of the surrounding HTML is a good idea in case you need to refer back to it when you’re crafting your regex.
Once you’re satisfied that we’re getting the correct HTML don’t forget to scroll to the bottom and click “Looks Good”.
You can check your regex, and we’ll show you what number we’re parsing.
Note that you must include at least one match group in your regular expression, as that is what we will parse out to use for the number to be entered on your goal.
If the regex is all good, then the rest of the setup process is much more familiar — you tell us units for what you’re tracking, and a rate for the red line, pick your initial safety buffer, and from there you’re done and we release you back into the regular flow of goal setup.
What about those examples you mentioned where Curlminder doesn’t work?
We’ll give you an example of that too:
That’s from Danny’s Fatebook profile. See that nice “They’ve made 7 forecasts” in the sidebar? Well, it’s nowhere to be found in the raw html. In the case of Fatebook (we know those guys) we’ll find a nice solution and hopefully make Fatebook an official integration before long. Unfortunately a lot of the web these days is constructed by javascript instructions after your browser has connected to the remote server, rather than the old fashioned way of having the remote send your browser an HTML document. That’s why we show you the html as we’ve fetched it during goal setup, so you can double check that your number is actually there, and your regex works.
Again, let us know if you run into a case like this where you can see the number on the rendered page but can’t find it in the html.
More use cases
Ok, we’ll close with a couple interesting use cases we’ve seen so far:
- Nicky is minding their number of minions in the game FFXIV
- User gbear605 is minding their
Mnemosyne progress via a couple API endpoints she contributed to the Mnemosyne server (when the contents of the URL is nothing but the number you care about, the regex to give Curlminder is just “
(.*)”) - Thanks to Theo Spears for pointing out that a URL like
leetcode-api-faisalshohag.vercel.app/USERNAMEcontains all manner of stats for Leetcode, if you want to beemind any of that via Curlminder
Footnotes
[1] Technically publicly accessible? Yes, like how Google Docs and Dropbox have a “share with link” option. That link is technically publically accessible but it includes a long unguessable string of characters so as long as the URL doesn’t fall into the wrong hands, the document is still effectively private.