[{"content":"How myriad.toml works, and how plugins read it.\nMyriad\u0026rsquo;s configuration model is smaller than it looks from the outside: one TOML file, and two ways to tell a plugin which section of it to read.\nThe myriad.toml file # Myriad looks for myriad.toml in the current directory by default. Each built-in plugin reads its settings from a named table in that file. For the fields plugin:\n[fields] namespace = \u0026#34;TestFields\u0026#34; namespace controls the namespace the generated module is emitted under; every built-in plugin currently exposes just that one key.\nPointing at a different config file # If you don\u0026rsquo;t want myriad.toml sitting in the project root, set MyriadConfigFile in the .fsproj:\n\u0026lt;PropertyGroup\u0026gt; \u0026lt;MyriadConfigFile\u0026gt;myConfig.toml\u0026lt;/MyriadConfigFile\u0026gt; \u0026lt;/PropertyGroup\u0026gt; Telling a plugin which config section to use # There are two ways to wire a generator to a config table, and they map to how the input is specified:\nAttribute-based: the normal case, when a plugin operates on annotated types in an F# source file. The string argument to the generator attribute is the config key:\n[\u0026lt;Generator.Fields \u0026#34;fields\u0026#34;\u0026gt;] type Test1 = { one: int; two: string; three: float; four: float32 } Here \u0026quot;fields\u0026quot; is both what tells the fields plugin which types in this file to process, and the [fields] table it reads from myriad.toml.\nMyriadConfigKey: used when a plugin\u0026rsquo;s input isn\u0026rsquo;t an attributed F# type at all (for example a plain text or data file), so there\u0026rsquo;s nowhere in the source to hang an attribute. The config key is set directly on the Compile element instead:\n\u0026lt;Compile Include=\u0026#34;ArbitaryFile.fs\u0026#34;\u0026gt; \u0026lt;MyriadFile\u0026gt;Test.txt\u0026lt;/MyriadFile\u0026gt; \u0026lt;MyriadConfigKey\u0026gt;example1\u0026lt;/MyriadConfigKey\u0026gt; \u0026lt;/Compile\u0026gt; This is how the Myriad.Plugin.Example1 sample plugin gets its namespace setting.\nHow a plugin actually reads it # Whichever mechanism supplied the key, the plugin receives it through GeneratorContext:\ntype GeneratorContext = { ConfigKey: string option ConfigGetter: string -\u0026gt; (string * obj) seq InputFilename: string ProjectContext: ProjectContext option AdditionalParameters: IDictionary\u0026lt;string, string\u0026gt; } ConfigKey is the string from either mechanism above; ConfigGetter is a function the plugin calls with that key to get back the matching table\u0026rsquo;s entries as (name, value) pairs. A plugin author never parses TOML directly; Myriad has already done that by the time Generate runs.\nOutput formatting # Generated code is formatted with Fantomas. If an .editorconfig file exists in or above the output file\u0026rsquo;s directory, Myriad reads its indent_size, max_line_length, end_of_line, insert_final_newline, and any fsharp_* properties and formats the generated file to match, so generated code follows the same style as everything else in the project without extra configuration. With no .editorconfig, Fantomas\u0026rsquo;s default style is used. The lookup is per output file, so different generated files in different directories can pick up different settings.\n","externalUrl":null,"permalink":"/projects/myriad/configuration/","section":"Projects","summary":"How myriad.toml works, and how plugins read it.\nMyriad’s configuration model is smaller than it looks from the outside: one TOML file, and two ways to tell a plugin which section of it to read.\nThe myriad.toml file # Myriad looks for myriad.toml in the current directory by default. Each built-in plugin reads its settings from a named table in that file. For the fields plugin:\n","title":"Configuration","type":"projects"},{"content":"The pipeline from annotated source to generated F#.\nMyriad is a small standalone CLI (Myriad.dll), driven either directly or via the MSBuild SDK, which shells out to that same CLI as a pre-build step. The pipeline is the same either way.\n1. Plugins are loaded, not linked # Myriad doesn\u0026rsquo;t ship generators built in: even the \u0026ldquo;built-in\u0026rdquo; fields/lenses/DuCases plugins are ordinary plugin assemblies (Myriad.Plugins.dll), loaded the same way a third-party plugin would be. Each --plugin \u0026lt;path-to-dll\u0026gt; argument is loaded via McMaster.NETCore.Plugins, and Myriad reflects over the loaded assembly for types carrying a generator-marker attribute. Since v0.8.5, the loader shares already-loaded assemblies like FSharp.Core and Fantomas.FCS with the plugin (PreferSharedTypes = true) rather than loading a second copy; that used to throw ReflectionTypeLoadException when a plugin was built against a slightly different target framework.\n2. One codegen unit at a time, one process for the whole project # A codegen unit is one input file with everything needed to process it: the file itself, an optional output file, an optional config key, any additional parameters, and an optional list of generator names to filter to. Units come from one of two places:\na single --inputfile/--outputfile pair, for direct CLI use, or a --manifest \u0026lt;file\u0026gt;.toml listing many units in one TOML file: this is what the MSBuild SDK uses. As of the 0.9.0 release, the whole project\u0026rsquo;s worth of files is processed in one Myriad.dll invocation instead of one process launch per attributed file, since the old per-file approach already had to reload plugins fresh every time anyway. 3. Each unit is run against every loaded generator # For every discovered generator type, Myriad checks whether its ValidInputExtensions includes the input file\u0026rsquo;s extension, and, unless the unit\u0026rsquo;s GeneratorFilters list excludes it by name, instantiates it and calls Generate with a GeneratorContext (see Configuration). A generator returns either Output.Ast (a list of F# AST module fragments) or Output.Source (a raw string); most built-in plugins return Ast. A generator can also implement IMyriadGeneratorWithDiagnostics instead of the plain interface, to report non-fatal warnings/info alongside its output rather than only being able to fail the whole build by throwing; these get printed as MSBuild-style diagnostic lines (path(line,col): severity CODE: message) so they surface the same way a compiler warning does.\nIf a generator throws, or reports an error-severity diagnostic, Myriad fails that unit with a formatted !CompilationError block rather than a raw exception dump, but other units in the same batch aren\u0026rsquo;t affected.\n4. AST becomes source, source gets formatted, formatting gets written # Output.Ast results are wrapped into a parse tree and handed to Fantomas (CodeFormatter.FormatASTAsync) to become source text; Output.Source results are used as-is. All of a unit\u0026rsquo;s generator outputs are concatenated, a fixed header comment is prepended (This code was generated by myriad. Changes to this file will be lost when the code is regenerated.), and the result is written to the unit\u0026rsquo;s output file, or, if MyriadInlineGeneration is set, spliced onto the end of the input file itself (via a temp file: read the input, strip anything after a previously-generated header if one\u0026rsquo;s already there, append the new generated block, then atomically replace the original).\n5. MSBuild integration is just automation around the same CLI # The Compile element\u0026rsquo;s MyriadFile/MyriadNamespace/MyriadConfigKey/MyriadInlineGeneration metadata gets collected into the manifest passed via --manifest, and an MSBuild Inputs/Outputs-driven target only re-runs generation when something\u0026rsquo;s actually changed: regenerating F# source is a normal part of the build, not a separate step you remember to run. --verbose and --wait-for-debugger are available for debugging the generator process itself, surfaced as MyriadSdkVerboseOutput/MyriadSdkWaitForDebugger MSBuild properties.\nSource: src/Myriad/Program.fs, src/Myriad.Core/Types.fs, and the project\u0026rsquo;s CHANGELOG.\n","externalUrl":null,"permalink":"/projects/myriad/how-it-works/","section":"Projects","summary":"The pipeline from annotated source to generated F#.\nMyriad is a small standalone CLI (Myriad.dll), driven either directly or via the MSBuild SDK, which shells out to that same CLI as a pre-build step. The pipeline is the same either way.\n","title":"How It Works","type":"projects"},{"content":"","date":"May 15, 2022","externalUrl":null,"permalink":"/","section":"7sharp9","summary":"","title":"7sharp9","type":"page"},{"content":"Follow my exploration and wittering on different aspects of programming here!\n","date":"May 15, 2022","externalUrl":null,"permalink":"/programming/","section":"Blog","summary":"Follow my exploration and wittering on different aspects of programming here!\n","title":"Blog","type":"programming"},{"content":"","date":"May 15, 2022","externalUrl":null,"permalink":"/tags/configuration/","section":"Tags","summary":"","title":"Configuration","type":"tags"},{"content":"","date":"May 15, 2022","externalUrl":null,"permalink":"/tags/fsharp/","section":"Tags","summary":"","title":"Fsharp","type":"tags"},{"content":"","date":"May 15, 2022","externalUrl":null,"permalink":"/tags/metaprogramming/","section":"Tags","summary":"","title":"Metaprogramming","type":"tags"},{"content":"","date":"May 15, 2022","externalUrl":null,"permalink":"/tags/myriad/","section":"Tags","summary":"","title":"Myriad","type":"tags"},{"content":"The other day I released version 0.8.1 of Myriad, its got some new configuration settings so I thought I would quickly talk about them here.\nNew Configuration Features # The main change in 0.8.1 is that it is now possible to specify filters so that only certain generators can run rather than all generators that are found withing a plugin or plugins that are present in your project file. This is done by adding the Generators element to your Myriad compile element within your project file.\nAdding a plugin # Briefly before we get onto the new configuration I mentioned, I just wanted to quickly cover the two way plugins are normally added to project. Plugins can be either enabled via a Nuget mechanism or by including an msbuild import that imports them. The Nuget method is what I would recommend as it means less editing of the project file and the plugin author would normally specify that it exports the plugin for Myriad to consume. The plugin author would include the following in a Myriad.Plugins.MyPlugin.props file in their project which would then be exported as part of the Nuget build:\n\u0026lt;Project\u0026gt; \u0026lt;ItemGroup\u0026gt; \u0026lt;MyriadSdkGenerator Include=\u0026#34;$(MSBuildThisFileDirectory)/../lib/net6.0/Myriad.Plugins.MyPlugin.dll\u0026#34; /\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;/Project\u0026gt; So when consuming the plugin via Nuget the msbuild property is included with your project and Myriad knows about the plugin assembly.\nIf you are testing or are not using Nuget for whatever reason then you can import the plugin manually into your project by adding the import yourself like this:\n\u0026lt;Import Project=\u0026#34;..\\Myriad.Plugins.Example1\\build\\Myriad.Plugins.Example1.InTest.props\u0026#34; /\u0026gt; This is mainly used for testing where you would not want to create a local nuget package just to test or debug something.\nWhen Myriad runs it searches for all plugins by looking at the assemblies specified by MyriadSdkGenerator and then finding all instances of types that implement the interface IMyriadGenerator.\n[\u0026lt;RequireQualifiedAccess\u0026gt;] type Output = | Ast of SynModuleOrNamespace list | Source of string type IMyriadGenerator = abstract member ValidInputExtensions : string seq abstract member Generate : GeneratorContext -\u0026gt; Output Any assemblies specified by MyriadSdkGenerator that have types that implement IMyriadGenerator are then ran and their generated output added to your project at the relevant places as specified by the configuration properties. I will go further into this aspect of configuration in a further post. Im currently working on full documentation but its a bit easier to write a series of blog posts as it forces you to start writing about all the little rabbit holes you fall down whole while describing the process, whereas documentation tends to read more linearly and implicit knowledge can sometimes be skipped over.\nThe Generators configuration # As you read above, all generators which implement IMyriadGenerator in any of the plugins are executed. This is normally ok but sometimes there might be a plugin which does not need file input and is driven by an external file or process, or there might be more than one plugin per assembly, this means that plugins have to be more defensively coded so they cannot be run if their relevant inputs are not satisfied. Its good practice to defensively code any plugins anyway as you would not want to cause an unrecoverable exception in the build process and instead want the plugin to fail gracefully. In the case of a Myriad plugin, if the plugin cannot run or the inputs result in an error, diagnostic output occurs and the plugin gracefully returns. This means Myriad can continue on with the next generators. The problem with this is its a little wasteful, ahead of compilation time you often know which plugins you want to run so the new configuration element Generators solves this issue.\nIf the Generators element is present then the semicolon separated list of filters are passed to Myriad so that only those named generators are ran. Lets look at the Generators element in a snipped of a build file:\n\u0026lt;Compile Include=\u0026#34;Input.fs\u0026#34;\u0026gt; \u0026lt;MyriadParams\u0026gt; \u0026lt;MyriadParam1\u0026gt;1\u0026lt;/MyriadParam1\u0026gt; \u0026lt;MyriadParam2\u0026gt;2\u0026lt;/MyriadParam2\u0026gt; \u0026lt;/MyriadParams\u0026gt; \u0026lt;Generators\u0026gt;LensesGenerator;FieldsGenerator;DUCasesGenerator\u0026lt;/Generators\u0026gt; \u0026lt;/Compile\u0026gt; This is part of the integration tests in Myriad, when Input.fs is compiled it is passed to any plugins that have generators. Within Myriad there are two plugin assemblies Myriad.Plugins.dll which contains the following generators: LensesGenerator, FieldsGenerator, and DUCasesGenerator. There is also another plugin assembly called Myriad.Plugins.Example1 which contains a generator called example1 which only operates on .txt extensions so when that plugin runs it wont have any valid input to generate for so it makes sense to exclude this plugin completely. Adding the element \u0026lt;Generators\u0026gt;LensesGenerator;FieldsGenerator;DUCasesGenerator\u0026lt;/Generators\u0026gt; allows you to pass this information to Myriad so ir will filter out all generators that are not named in the Generators element.\nThe new configuration gives you more flexibility in controlling which plugins do and don\u0026rsquo;t get run and as a result Myriad will be faster and more efficient as it will get run for the generators specified in the filter. Don\u0026rsquo;t worry though if the Generators element is not present then all generators will get run as usual.\nI hoped you enjoyed reading a little about the new configuration and a small about of background relating to it.\nThanks for reading, Until next time!\n","date":"May 15, 2022","externalUrl":null,"permalink":"/programming/2022-05-15-myriad-configuration-changes/","section":"Blog","summary":"The other day I released version 0.8.1 of Myriad, its got some new configuration settings so I thought I would quickly talk about them here.\n","title":"Myriad Configuration Changes","type":"programming"},{"content":"","date":"May 15, 2022","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":" Hi, I\u0026rsquo;m Dave Thomas, a software engineer from the UK, writing primarily in F# with a fondness for Rust and C++. I\u0026rsquo;m drawn to functional languages and the kinds of problems that sit at the intersection of language design, tooling, and performance.\nOver the years I\u0026rsquo;ve contributed to the F# ecosystem: co-created FSharp.Compiler.Service, built Myriad (a metaprogramming and code generation tool for F#), and spent a lot of time making F# work well on platforms it wasn\u0026rsquo;t originally designed for.\nI\u0026rsquo;m also an MSc Artificial Intelligence student, which has pulled my interests further into compilers, DSP and reverse engineering: the common thread is always what\u0026rsquo;s happening underneath the abstraction.\nWhen I\u0026rsquo;m not writing code I\u0026rsquo;m usually making too much noise on the guitar.\nFeel free to ping me on Bluesky.\n","date":"October 15, 2020","externalUrl":null,"permalink":"/about/","section":"7sharp9","summary":" Hi, I’m Dave Thomas, a software engineer from the UK, writing primarily in F# with a fondness for Rust and C++. I’m drawn to functional languages and the kinds of problems that sit at the intersection of language design, tooling, and performance.\n","title":"About","type":"page"},{"content":"","date":"August 13, 2020","externalUrl":null,"permalink":"/tags/exercise/","section":"Tags","summary":"","title":"Exercise","type":"tags"},{"content":"","date":"August 13, 2020","externalUrl":null,"permalink":"/tags/fitness/","section":"Tags","summary":"","title":"Fitness","type":"tags"},{"content":"Running is not something Ive ever posted about before but it\u0026rsquo;s been a part of my life for a long time.\nI normally mainly just write about programming but over the years Ive learned that writing in more general terms about life and experiences gives you more of an insight into a person and makes any subsequent material more relateable, although I\u0026rsquo;ve not put that into practice much its something Im trying to do more of starting with this post.\nWhere did running begin? # Originally I started running because I was training for a job in the armed services, but things changed and that didn\u0026rsquo;t end up happening. What did change though was that I found a love/hate relationship with running. Well not love hate really, more that it\u0026rsquo;s a mental battle between your conscious and subconscious mind. Your conscious mind is telling you, you can do this, while your subconscious mind is telling you that you are suffocating, you need more air you have to stop!\nWhy Running? # What I really like about running is it both a challenge and also gives you great sense of well being, it\u0026rsquo;s great for clearing mental fog and reducing stress. Contrary to popular belief if you run correctly it can strengthen your knees and ligaments, help to build stronger bones, improve cardiovascular fitness and maintain a health weight. There is some research that shows running can boost your memory, improve focus, and fight depression and anxiety. I find I have a lot more clarity and calmness after a run and things I would of found stressful in programming do not bother me as much.\nSince we moved house 3 years ago I\u0026rsquo;ve been training more seriously on and off, moving house does cause a lot of disruption so it\u0026rsquo;s taken a while to get back into it again. Just before lockdown started in the UK back in March this year (2020) we bought a treadmill so that we could run or walk to get a workout without having to venture outside as much and bump into anyone as much. The treadmill we bought was Nordic Track which has a LCD screen and iFit integrated. I immediately found the trainer @tommy_rivs, he is a really good iFit trainer, very charismatic person and keeps you motivated throughout the training, as well as importing knowledge on physiology and culture, no matter where he is running you will learn something new as well as having a good workout!\nI follow @tommy_rivs on social media and recently found out he had been hospitalized, he was then diagnosed with primary pulmonary NK/T-cell lymphoma, this is a rare type of lung cancer. He’s a pro runner from Flagstaff Arizona, physical therapist, and well known in the running community. At first I found out about a fund raiser by iFit where they would donate $1 for every mile completed on one of his programs, so my immediate thought is I would definitely take part in that and crank out as many miles as I could. Later that day I think, I learned about the #RunWithRivs challenge where you set yourself a challenge then use that positivity and motivation to raise funds for Tommy to help with medical costs. So that was that, I set myself a challenge of running 45 miles during the next week. Im more of a 5 to 10k runner so I though this might be possible for me although I knew I might be a bit sore.\nResults # I ended up running 58 miles during the challenge and raising $115, although i didn\u0026rsquo;t quite meet my target of $150 I really glad I did the challenge. I\u0026rsquo;m just glad I could help even if it\u0026rsquo;s just in some small way. If you want to check out my challenge page then it\u0026rsquo;s here it\u0026rsquo;s still open for donations if you wanted to help out.\nHeres me after completing the challenge last Sunday :-)\nWow! I was tired but it was worth it! It\u0026rsquo;s given me the bug for doing longer runs and challenges in the future. I would love to do an ultra marathon!\nAs for the challenge as a whole, over $180,000 was raised!\nOh finally, iFit recently announced the outcome of the fund raiser where they were donating $1 per mile.\nThe participants came from 81 different countries, completed 32,934 workouts and covered 106,751 miles, so that mean iFit are donating $106,751! Amazing!\nUntil next time! # ","date":"August 13, 2020","externalUrl":null,"permalink":"/running/2020-08-04-run-for-your-life/","section":"Runnings","summary":"Running is not something Ive ever posted about before but it’s been a part of my life for a long time.\n","title":"Run for Your Life","type":"running"},{"content":"","date":"August 13, 2020","externalUrl":null,"permalink":"/running/","section":"Runnings","summary":"","title":"Runnings","type":"running"},{"content":"So Ive had my YouTube channel for a year now, how did it go?\nAnalytics numbers and figures # 17 Videos created 755 Subscribers 15.1k Total views 693.2 Hours viewing time Most popular video is an introduction to meta-programming with F# and Quotations:\nFollowed close behind by: F# Platform Game Series:- Getting Started with F# And MonoGame.\nThe most popular overall topic was games development which I can understand as its a fun topic with a strong visual response to keep interest. There\u0026rsquo;s also a vast cornucopia of topics that you could put in games development video so the area is huge to explore.\nAs usual with any analytics you end up wanting to reach milestones, I would of loved to get to 1000 subscribers figure even though that serves no purpose other than being a nice whole number. It is the entry requirement to YouTube monetization of a channel along with 4000 hours viewing time. To get my channel monetized I would need 145 more subscribers and a whopping 3306.8 more hours viewing time! Lets not dwell on the figures as its depressing, meaningless and not really healthy.\nEquipment # So what did I use over the course of the year and how did it evolve?\nCamera # I started out by making a test video using my iPad Pro balanced on a cardboard box using no external lighting, no microphone. Needless to say I decided to invest in some equipment to try and make videos that were at least semi decent quality. My wife is an illustrator so she had some camera equipment and lighting, so I borrowed her Nikon D7000 to try out, using that to record video. It turns out that the autofocus was extremely poor so I decided to invest in a Canon G7X Mark II. Its a really good vlogging camera as I found a number of people recommending it for YouTube.\nCurrently using\nCanon G7X Mark III\nWishlist Camera # My ultimate camera could be a Nikon Z6 or a Canon EOS RP, either of those is on my wish list for making epic b-roll and talking head shots.\nLighting # Now on to lighting, it pretty much essential for talking head videos unless you have some strong natural light to work with. I used one I made myself using a circular metal foiled cardboard ring with flexible LED strip lights attached, or I borrowed my wife\u0026rsquo;s ring light.\nCurrently using\nNanGuang CN-R640 19\u0026quot; Outer Photography Video Studio 640 LED CRI 95 5600K Dimmable Ring Light - 3850 lumens\nWishlist Lighting # An Apature LS C120d II with a Light Dome would be an incredible light, expensive but versatile and awesome.\nMicrophone # A microphone is necessary, no way around it, good sound is essential, don\u0026rsquo;t scrimp on quality either and learn about the different types of microphone available and placing the microphone correctly. I opted for a Rode NTG4 shotgun microphone mounted on a microphone tripod boom arm just out of view the top of the shot, probably around 40cm diagonally up from my mouth. If I was moving about more I would of used a lavalier microphone, if I was doing hand held vlogging then a camera mounted mic like a Rode Video Mic Pro+ would of been ideal.\nCurrently using\nRode NTG4 shotgun microphone\nWishlist Microphone # Im pretty happy with Rode NTG4 but given the chance I might get a RODE Video Mic Pro+ If I ever got a DSLR from my wish list! I would also like to try out the more expensive Rode NTG3 Shotgun Mic\nStorage # When I first started making videos storage didn\u0026rsquo;t even occur to me, but when you make even small 10 minute vies the storage space needed for efficient editing can be huge especially if you transcode to a format suited to fast video scrubbing so that the CPU doesn\u0026rsquo;t have to bust a gut like ProRes, which requires 882.15 MB a minute for 1080p 23.076 footage.\nI opted for a fast SSD to store and edit my footage on\n2TB SanDisk Extreme SSD Media\nWishlist storage # I think I would go for some sort of SSD raid configuration with backups to another larger RAID device, at least 50TB, video just eats storage.\nSoftware # I have used pretty much all of the top video editors that are available, and I can pretty much say I can break them all with ease!\nApple Final Cut Pro # I started out with Final Cut Pro as I was most familiar with that as I had helped my wife with some parts of her art videos during Inktober 2018, which then inspired me to start my own YouTube channel. It\u0026rsquo;s pretty easy to learn and get efficient at editing. The magnetic timeline feature is really nice to work with. One of the things that I think limits Final Cut Pro a little is that some more advanced effects and editing seem to be more limited or tricky in Motion, which is the Apple comparative to Adobe\u0026rsquo;s After Effects.\nAdobe Premiere # When i started to use Patreon I decided to invest in Adobe Creative Cloud which included Premier, After Effects and most of the other design oriented products. As there was a lot more tutorials and professional plugins available for Premier and After Effects I thought it would be interesting to see what I could do. All in all I was pretty happy with everything except the cost. I could live with the crashes as everyone seem to do with Premiere but the cost was the biggest issue for me. I was basically using my Patreon funding for this, investing the surplus into saving up for new gear.\nDaVinci Resolve 16 # I evaluated DaVinci Resolve 16 when I was looking at Premiere and found I could do similar things to both Final Cut Pro and Premiere but the interface was a lot more clunky and the polish of the software was not in the same league as Final Cut Pro or Adobe Premier. Also documentation and tutorials were thin on the ground compared to the plethora available for Premiere and Final Cut Pro. The node based effects system is interesting but takes a while to get used to, it\u0026rsquo;s somewhat reminiscent of SoftImage. It seems to be a lot more time consuming than using After Effects which also has magnitudes more tutorials, documentation and plugins available too.\nSuccessful? # I think all in all its been a successful year, I managed to make a sponsored video for GitPod and I also did some further client video work so it has open up some new avenues to me. I learned a whole lot about the creative process of making videos, editing skills and software and hardware knowledge thats needed for video creation.\nI think thats enough for now but if anyone is interested I might follow this up with how my workflow has evolved over the last year too, let me know in the comments below!\nUntil next time!\n","date":"December 3, 2019","externalUrl":null,"permalink":"/programming/2019-12-03-one-year-on-youtube/","section":"Blog","summary":"So Ive had my YouTube channel for a year now, how did it go?\n","title":"One year on YouTube","type":"programming"},{"content":"","date":"December 3, 2019","externalUrl":null,"permalink":"/tags/videos/","section":"Tags","summary":"","title":"Videos","type":"tags"},{"content":"","date":"December 3, 2019","externalUrl":null,"permalink":"/tags/youtube/","section":"Tags","summary":"","title":"Youtube","type":"tags"},{"content":"","date":"November 6, 2019","externalUrl":null,"permalink":"/tags/ast/","section":"Tags","summary":"","title":"Ast","type":"tags"},{"content":"I released a new meta programming club video yesterday on my YouTube channel\nI thought I would write a little about it here on my blog too.\nI recently did a little work to flesh out the plugin interface for Myriad. My friend Enrico Sada helped out with some of the fiddly MsBuild work in dotnet which can be a little confusing at times.\nThe gist of the plugin is that you must implement the following interface:\ntype IMyriadGenerator = abstract member Generate: namespace\u0026#39;: string * ast:ParsedInput -\u0026gt; SynModuleOrNamespaceRcd And annotate you type with the MyriadGeneratorAttribute specifying the name of your plugin so that it can be found.\nHeres an example plugin from the Myriad repo:\n[\u0026lt;MyriadGenerator(\u0026#34;example1\u0026#34;)\u0026gt;] type Example1Gen() = interface IMyriadGenerator with member __.Generate(namespace\u0026#39;, _ast) = let let42 = SynModuleDecl.CreateLet [ { SynBindingRcd.Let with Pattern = SynPatRcd.CreateLongIdent(LongIdentWithDots.CreateString \u0026#34;fourtyTwo\u0026#34;, []) Expr = SynExpr.CreateConst(SynConst.Int32 42) } ] let componentInfo = SynComponentInfoRcd.Create [ Ident.Create \u0026#34;example1\u0026#34; ] let nestedModule = SynModuleDecl.CreateNestedModule(componentInfo, [ let42 ]) let namespaceOrModule = { SynModuleOrNamespaceRcd.CreateNamespace(Ident.CreateLong namespace\u0026#39;) with Declarations = [ nestedModule ] } namespaceOrModule All this example does is generate a simple module like this:\nmodule example1 = let fourtyTwo = 42 Ast Helping Hand # Unfortunately working with the AST can be quite verbose in Myriad I use FsAst which uses the record update syntax to aid in the construction of AST nodes by using the record update syntax so you don\u0026rsquo;t have to supply every parameter.\nTake a Let binding AST element:\n{SynBindingRcd.Let with Pattern = pattern Expr = expr } If we did not use the record update syntax in SynBindingRcd then we would have to supply every parameter for the let binding:\n{ Access = None Kind = SynBindingKind.NormalBinding IsInline = false IsMutable = false Attributes = SynAttributes.Empty XmlDoc = PreXmlDoc.Empty ValData = SynValData(Some MemberFlags.InstanceMember, SynValInfo.Empty, None) Pattern = pattern ReturnInfo = None Expr = expr Range = range.Zero Bind = SequencePointInfoForBinding.NoSequencePointAtInvisibleBinding } Which is not exactly fun, the record update syntax makes these type of things easy to define and compose together.\nThe new version of Myriad can be found on Nuget here or at its repo.\nI hope you enjoyed the video and this brief blog!\nUntil next time!\n","date":"November 6, 2019","externalUrl":null,"permalink":"/programming/2019-11-06-myriad-intro/","section":"Blog","summary":"I released a new meta programming club video yesterday on my YouTube channel\nI thought I would write a little about it here on my blog too.\nI recently did a little work to flesh out the plugin interface for Myriad. My friend Enrico Sada helped out with some of the fiddly MsBuild work in dotnet which can be a little confusing at times.\n","title":"Myriad Intro","type":"programming"},{"content":"","date":"November 6, 2019","externalUrl":null,"permalink":"/tags/quotations/","section":"Tags","summary":"","title":"Quotations","type":"tags"},{"content":"A code generation and meta-programming tool for F#.\nMyriad is a code generation and meta-programming tool for F#, built to make compile-time codegen a first-class, idiomatic part of the language rather than a workaround.\nF# doesn\u0026rsquo;t have macros, and Type Providers solve a narrower problem than general-purpose code generation. Myriad fills that gap: annotate a type, point it at a plugin, and it emits real F# (discriminated unions, records, lenses), generated before compilation and optimized like everything else. No reflection at runtime, nothing at the type level you can\u0026rsquo;t inspect.\nIt ships with three plugins out of the box: fields (inspired by OCaml\u0026rsquo;s ppx_fields_conv), lenses (functional getter/setter pairs), and DuCases (toString/fromString/toTag/isX helpers for discriminated unions). It runs via MSBuild or as a standalone CLI, and is extensible with your own plugins for whatever pattern you\u0026rsquo;re tired of hand-writing. It\u0026rsquo;s Apache-2.0 licensed, actively maintained (it passed 1.0 in September 2026, making the public API stable), and has grown a small ecosystem of third-party plugins (SqlHydra, JsonWrapper, TypeSafeInternals among them).\n\u0026lt;PackageReference Include=\u0026#34;Myriad.Core\u0026#34; /\u0026gt; \u0026lt;PackageReference Include=\u0026#34;Myriad.Sdk\u0026#34; /\u0026gt; [\u0026lt;Generator.Fields \u0026#34;fields\u0026#34;\u0026gt;] type Test1 = { one: int; two: string } Source, releases, and the changelog are on GitHub. For the mechanics, see Configuration for myriad.toml and how plugins receive settings, and How It Works for the CLI/MSBuild pipeline itself.\nFurther reading, from when I was building it:\nApplied Meta-Programming with Myriad: the original writeup, with Falanx Myriad Intro: a Meta Programming Club talk introducing it Myriad Configuration Changes: the most recent update to the config system ","date":"November 6, 2019","externalUrl":null,"permalink":"/projects/myriad/","section":"Projects","summary":"A code generation and meta-programming tool for F#.\nMyriad is a code generation and meta-programming tool for F#, built to make compile-time codegen a first-class, idiomatic part of the language rather than a workaround.\nF# doesn’t have macros, and Type Providers solve a narrower problem than general-purpose code generation. Myriad fills that gap: annotate a type, point it at a plugin, and it emits real F# (discriminated unions, records, lenses), generated before compilation and optimized like everything else. No reflection at runtime, nothing at the type level you can’t inspect.\n","title":"Myriad","type":"projects"},{"content":"A few of the things I\u0026rsquo;ve built and kept maintaining.\n","date":"November 6, 2019","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"A few of the things I’ve built and kept maintaining.\n","title":"Projects","type":"projects"},{"content":"Ive decided that this month is last month I\u0026rsquo;ll be on Patreon.\nIve decided at the end of this month I\u0026rsquo;m going to stop using Patreon.\nI’ve come to this decision for several different reasons which I will go into now.\nFirstly my YouTube channel has now become my main creative outlet. By that I mean none of my other hobbies even get a chance, I\u0026rsquo;m always thinking about making a new video mainly because of the subscription based nature of Patreon makes it feel like you have to have content month on month without fail. I started out making a video a week at the beginning but I quickly realised this is not sensible with the level of production and quality that I like to release, so once every 3 to 4 weeks has become the norm now. Definitely every month now that I have patrons waiting for content.\nI don’t think that I\u0026rsquo;m going to stop making videos I really like making videos, I just don’t want the pressure of Patreon to be something that influences my decisions going forward. Even if it is irrational, some of you might say \u0026ldquo;we don’t want to put you under pressure we just want to support you\u0026rdquo;. While I am really grateful for the community we have started here, and I hope if can continue without Patreon.\nI want the videos I produce to maintain the same quality of production. I would like to take that even further in the future if my channel ever got to the point where I could upgrade my video equipment to a nice DSLR and a 4k monitor so I could record and produce full 4k videos. (Currently I upscale from 1080p/2k screen casts to 4k as it gets rendered better by YouTube).\nI have been spending the Patreon money mainly on an Adobe Creative Cloud subscription and an Epidemic sounds subscription as I wanted to see if I could use software like Adobe Premiere so increase the level of production further. While I feel like the quality has increased I don’t think its due to the software, its more about better sound and video editing, I think thats made the biggest difference. Another reason I chose to use Adobe Premier was the bugs I was previously getting in Final Cut Pro, however, Adobe Premier Pro has been diabolical this year so I\u0026rsquo;m going back to Final Cut Pro. So with the end of Patreon funding I\u0026rsquo;m also stopping Creative cloud and Epidemic sounds so my videos may look and sound a little bit different from now on. For a small channel like mine its quite an expense to fund music licensing with Epidemic Sounds and a Creative Cloud subscription service each month.\nThere\u0026rsquo;s also some other things that have come into play. I\u0026rsquo;m a freelance programmer and since April I\u0026rsquo;ve taken a break and have been doing a little research into new languages and ideas as well as planing and recording new videos. I\u0026rsquo;ve now got to the point where I need to fund my business again with more freelance work so that means less time to commit to YouTube and it doesn’t seem right to expect the Patreon funding to continue, and also for the reasons I mentioned previously about feeling the pressure to create each month without fail.\nI\u0026rsquo;m still going to keep my ko-fi account that I recently setup for ad-hoc support of my channel which I’ll continue to use to help fund future equipment like a better camera, lighting, and sound etc. But this will be more along the lines of:\nIf you liked this video and want to support the future production of videos then why not buy me a coffee!\nko-fi.com/7sharp9\nAgain I want to thank everyone for supporting me over the last 6 months on Patreon!\nThank you!\n7sharp9.\n","date":"October 21, 2019","externalUrl":null,"permalink":"/programming/2019-10-21-final-month-on-patreon/","section":"Blog","summary":"Ive decided that this month is last month I’ll be on Patreon.\n","title":"Final Month On Patreon","type":"programming"},{"content":"This article is all about F# metaprogramming and research that I did which evolved over time, firstly with a project called Falanx that I developed while working for Jet/Walmart, then later the ideas and concepts evolved into Myriad.\nWhat is Myriad? # Myriad is a pre-compilation tool that generates code from code. It is integrated into the project build cycle via an MSBuild extension. It is also possible to invoke the tool separately as a CLI tool but this essay will deal with describing the integration within an MSBuild project as this would be the most common pattern of usage.\nHistorical basis # Before I dig into the detail its best to get some historical details as the Myriad project stems from previous meta-programming work that I have been doing over the last year, perhaps also further back through my history of using F# too. The Falanx project built the first steps towards Myriad by taking blocks of functionality from different areas of the F# ecosystem and welding them together to aid in code generation. Lets go through some of the different way you can utilise meta-programming in F#.\nMeta-programming 101 # F# has a number of meta-programming facilities that I have spoke about before in my previous blog posts: Code Quotations, Type Providers, Typed Expressions, and also the untyped abstract syntax tree - AST.\nQuotations # Quotations are backed by reflection and mainly used to transform F# to another language. They are limited in that they do not represent the whole F# language. Types and modules are not able to be represented like they are in the F# AST. They also do not encode F# on a one to one basis in terms of representing F# expressions. Some elements like discriminated union decomposition, pattern matching and function fixity are not represented in the same form as they occur in F#, details are lost in transformation.\nType Providers # Type Providers use Quotations to encode method information and use these expression with a skeleton of types produced by some for of schema or input. Type Providers can be useful in some limited scenarios. Building a Type Provider should not be taken lightly as there can often be a lot of edge cases and debugging before they are production ready. There\u0026rsquo;s also the fact that Type Providers can not create any F# constructs like records or Discriminated Unions.\nTyped Expressions # Typed expressions are used for whole language or system transformations and is the technique used in Fable to transpile F# to JavaScript. You can read more about typed expressions in my Metamatic blog post. They have no dependency on reflection and do not require any on disk assemblies.\nUntyped AST # The untyped AST or AST for short is not an area that has been widely used, mainly as working with the AST is quite tricky outside of the compiler and the AST is normally produced by the compiler as a result of the parsing phase.\nFalanx Whistle-stop Tour # It is not the scope of this essay to describe Falanx in detail but reading some of the background on it may provide an insight on how the direction in Myriad was formed and how some of the technical challenges experienced in Falanx drove the resulting design decisions. The following is a quick run through some of the core concepts and issues faced in Falanx.\nFalanx uses a mixture of Type Providers, (mainly the Type Provider SDK not the actual Type Provider invocation mechanism) Quotations, and AST manipulation to generate source code. Falanx evolved from a minimum viable product where the key input factors were a prototype which could be developed within a few weeks that could take a protocol buffer schema and generate F# records and Discriminated Union types as output.\nThis is an example of a `Protocol Buffer file:\nsyntax = \u0026#34;proto3\u0026#34;; message BundleRequest { int32 martId = 1; string member_id = 2; } Falanx works by defining an MSBuild extension that references a Protocol Buffer file and generates another file in response to it containing F# records and discriminated unions.\n\u0026lt;ItemGroup\u0026gt; \u0026lt;PackageReference Include=\u0026#34;Falanx.Sdk\u0026#34; Version=\u0026#34;0.4.*\u0026#34; PrivateAssets=\u0026#34;All\u0026#34; /\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;ProtoFile Include=\u0026#34;..\\proto\\bundle.proto\u0026#34;\u0026gt; \u0026lt;OutputPath\u0026gt;mycustom.fs\u0026lt;/OutputPath\u0026gt; \u0026lt;/ProtoFile\u0026gt; \u0026lt;ProtoFile Include=\u0026quot;..\\proto\\bundle.proto\u0026quot;\u0026gt; is the input file and \u0026lt;OutputPath\u0026gt;mycustom.fs\u0026lt;/OutputPath\u0026gt; is the output file.\nThe resulting generated source code looks like this:\n[\u0026lt;CLIMutable\u0026gt;] type BundleRequest = { mutable martId : int option mutable memberId : string option } static member JsonObjCodec = fun martId memberId -\u0026gt; { martId = martId memberId = memberId } \u0026lt;!\u0026gt; Operators.jopt\u0026lt;BundleRequest, Int32\u0026gt; (\u0026#34;martId\u0026#34;) (fun x -\u0026gt; x.martId) \u0026lt;*\u0026gt; Operators.jopt\u0026lt;BundleRequest, String\u0026gt; (\u0026#34;memberId\u0026#34;) (fun x -\u0026gt; x.memberId) static member Serialize(m : BundleRequest, buffer : ZeroCopyBuffer) = writeOption\u0026lt;Int32\u0026gt; (writeInt32) (1) (buffer) (m.martId) writeOption\u0026lt;String\u0026gt; (writeString) (2) (buffer) (m.memberId) static member Deserialize(buffer : ZeroCopyBuffer) = deserialize\u0026lt;BundleRequest\u0026gt; (buffer) interface IMessage with member x.Serialize(buffer : ZeroCopyBuffer) = BundleRequest.Serialize(x, buffer) member x.ReadFrom(buffer : ZeroCopyBuffer) = let enumerator = ZeroCopyBuffer.allFields(buffer).GetEnumerator() while enumerator.MoveNext() do let current = enumerator.Current if current.FieldNum = 2 then x.memberId \u0026lt;- (Some(readString current) ) else if current.FieldNum = 1 then x.martId \u0026lt;- (Some(readInt32 current) ) else () enumerator.Dispose() member x.SerializedLength() = serializedLength\u0026lt;BundleRequest\u0026gt; (x) The first part of the generated code is a record definition followed by binary and json serialization methods. JsonObjCodec is used via the Fleece library, Serialize and Deserialize are used by the Froto12 library. The records produced support both binary and json serialization via the Froto and Fleece libraries respectively. Fleece ss interesting in that the generated code uses applicative style and only requires a codec property JsonObjCodec to serialize and deserialize to json:\n//serialize let bundleRequest = {name = Some 42; memberId = Some \u0026#34;172c\u0026#34; } printfn \u0026#34;%s\u0026#34; (string (toJson bundleRequest)) //deserialize let bundleRequest2 = parseJson\u0026lt;BundleRequest\u0026gt; \u0026#34;\u0026#34;\u0026#34;{\u0026#34;martId\u0026#34;: \u0026#34;54\u0026#34;, \u0026#34;memberId\u0026#34;: \u0026#34;172d\u0026#34;}\u0026#34;\u0026#34;\u0026#34; Parsing/AST # Part of the technical challenge of Falanx was the time pressure to build a minimum viable product in a short amount of time. Type Providers were out of the question as one of the main requirements were to produce F# records and Discriminated Unions.\nThe Froto library already had a working Type Provider but the resulting output code was not F# records and [Discriminated Unions22 as Type Providers only support basic CLR types no F# specific types are supported. Another issue was that Protocol Buffers version 2 was the only version supported, so we decided to use the parser from Froto and make a PR to update it to support Protocol Buffers 3 syntax.\nQuotations # We now have the Protocol Buffer 3 file represented as an abstract syntax tree. In Froto there also exist quotations that were used for the Provided Type methods that were used in the Froto Type Provider, although we were not using the [Type Provider9 in Froto, we could reuse some of these quotations and adapt them for our needs. Extra quotations were also created to form the JsonObjCodec property from the code above.\nQuotations -\u0026gt; AST # The next part was to take the quotations representing Serialize, Deserialize and JsonObjCodec and convert them to source code. Both quotations and the F# AST represent similar but not quite the same things: A collection of nodes that represent the abstract notion of code. Quotations do not map fully into the F# AST as they only represent a subset of the AST, types for example are not present in quotations, neither can quotation map functions like for like into F# AST nodes as quotations lack detail compared the the F# AST, such as fixity information, pattern matching and decomposition of Discriminated Unions are all altered in the quotation of literals or composing of Quotation Expressions.\nTheres an interesting library called Quotation Compiler by Eirik Tsarpalis, inside this library there a piece of code which uses an entry point into the F# compiler that allows you to compile an AST to a dll rather than using a source file. I remember looking through this previously and wondered if something similar could be used. I also found that there was a function that transformed quotations to fragments of an AST too. I found that this could be adapted to what I needed with some simple changes. Unfortunately it was not possible to reference this library directly as in the end I needed to heavily modify it to work with the quotations that referenced Provided types due to reflection issues, but it did form the basis for the bulk of the solution. The reflection issue it to do with the way Type Providers are represented. Each type produced in a generative type provider is backed by a subtype of one of the reflection base types:\nType Base ProvidedTypeSymbol TypeDelegator ProvidedSymbolMethod MethodInfo ProvidedStaticParameter ParameterInfo ProvidedParameter ParameterInfo ProvidedConstructor ConstructorInfo ProvidedMethod MethodInfo ProvidedProperty PropertyInfo ProvidedEvent EventInfo ProvidedField FieldInfo ProvidedTypeDefinition TypeDelegator MethodSymbol2 MethodInfo ConstructorSymbol ConstructorInfo MethodSymbol MethodInfo PropertySymbol PropertyInfo EventSymbol EventInfo FieldSymbol FieldInfo Type Symbol TypeDelegator TargetGenericParam TypeDelegator TargetTypeDefinition TypeDelegator TargetModule Module TargetAssembly Assembly ProvidedAssembly Assembly These can start to become an issue when you start to use reflection on the Provided Types as the Type Provider SDK only implements the minimal overloads from the base types so you can quickly encounter not implemented exceptions quite easily. An example of this is if you tried to access the ReflectedType property of a ProvidedProperty\nlet notRequired this opname item = let msg = sprintf \u0026#34;The operation \u0026#39;%s\u0026#39; on item \u0026#39;%s\u0026#39; should not be called on provided type, member or parameter of type \u0026#39;%O\u0026#39;. Stack trace:\\n%s\u0026#34; opname item (this.GetType()) Environment.StackTrace Debug.Assert (false, msg) raise (NotSupportedException msg) type ProvidedProperty(...) = inherit PropertyInfo() ... override this.ReflectedType = notRequired this \u0026#34;ReflectedType\u0026#34; propertyName This ended up requiring some creative workarounds and reverse engineering of the reflection functionality in FSharp.Core which was quite time consuming. An example of this when converting a NewRecord quotation expression inot an aST fragment:\nmatch expr with | NewRecord(ty, entries) -\u0026gt; let synTy = sysTypeToSynType range ty knownNamespaces ommitEnclosingType let fields = match ty with | :? ProvidedRecord as pr -\u0026gt; pr.RecordFields | _ -\u0026gt; FSharpType.GetRecordFields(ty, BindingFlags.NonPublic ||| BindingFlags.Public) |\u0026gt; Array.toList let synEntries = List.map exprToAst entries let entries = (fields, synEntries) ||\u0026gt; List.map2 (fun f e -\u0026gt; (mkLongIdent range [mkIdent range f.Name], true), Some e, None) let synExpr = SynExpr.Record(None, None, entries, range) SynExpr.Typed(synExpr, synTy, range) If you look at match ty with you can see that theres a special pattern match when the ty is a ProvidedRecord which calls pr.RecordFields instead of FSharpType.GetRecordFields. This is because GetRecordFields accesses reflection data that is not implemented fully in the Type Provider SDK this is due to custom attributes that FSharp.Core expect to be present.\nAnother aspect of this was transforming ProvidedTypes into AST nodes. As well as the built in types in the Type Provider SDK I added several new types namely a ProvidedUnion, ProvidedUnionCase and a ProvidedRecord. These custom types along with normal Provided Types were mapped into AST fragments along with the quotations that had been transformed into AST nodes. The combination of all these elements were used to create a complete AST which comprised of Modules, Records with member functions and Discriminated Unions with member functions.\nAST -\u0026gt; Source code # We did not need the output to be a dll but actual source code as this was another of the main requirements, to do this I used the Fantomas library which allows you to use an AST as input and get back the source code the AST represents.\nSummary of Challenges with Falanx # Theres were quite a few challenges with Falanx, mainly around the usage and consumption of quotations. In other languages quoting and unquoting is a first class part off the language, however, this is not so with F# so there are lots of pitfalls whilst working with quotations.\nQuotations are really difficult to work with when you need to compose complex functions, even more so if you mix in Statically Resolved Type Parameters and reflection.\nQuotations loose detail when you use quotation literals (sections of code enclosed by the \u0026lt;@/@\u0026gt; and \u0026lt;@@/@@\u0026gt; operators). For example pattern matching gets transformed into if else blocks, fixity information is lost so you don\u0026rsquo;t know if an operator was called with infix or prefix notation etc. (I raised some of these issues as F# language suggestions: Add union match patterns to Expr, Add support for recording fixity in quoted literals)\nQuotations do not represent the full range of expressions possible with F#. Types and Modules are not representable, Discriminated Union decomposition has no form of Quotation.\nThe transformed quotations are not always elegant idiomatic F# code, mainly for the reason above and the fact that pattern matching is erased etc.\nSplicing types into quotations can be really tricky especially if you are using a library that uses a lot of generic parameters such as Fleece. Type inference in a library like Fleece makes it really nice to use with custom applicative operators but defining generic functions with 5 or more generic types in the type signature is not fun at all. Theres is an approved language proposal to add the splicing of types with the ~ operator which would help with this somewhat.\nWorking with Provided Types from the Type Provider SDK can be really challenging with quotations as quotations are backed by reflection and Provided Types have bare minimum reflection implemented. There are often times where composing Quotations will fail at runtime due to a missing reflection implementation. This sometimes required a patch to either the reflection implementation in the Type Provider SDK or a custom reflection implementation to extract information from the Provided Types.\nAll in all it was quite an intense development processes with lots and lots of debugging and digging through FSharp.Core code and coming up with creative solutions (Hacks!) to reflection issues. I could probably write an entire article all about the pitfalls, tricks and tips of working with quotations without shooting your self in the foot with a blunderbuss.\nMyriad Technical Overview # Ok enough with Falanx what is Myriad Again? # Myriad is similar to the ppx_deriving extension for OCaml except that it\u0026rsquo;s not as integrated as OCaml. Although F# compiler integration is desirable for a tool like this there is the question of whether or not this would be in scope for the future direction of the F# language. There is also the latency factor of writing and extension for the F# compiler and waiting for a release cycle so that it becomes generally available for everyone to use. By leveraging MSBuild its possible to come close to some of the capabilities and functionality of ppx_deriving. At the very least it progresses the notion of what more advanced macro like capabilities of F# could look like.\nppx_deriving works by allowing a type to extended by an arbitrary function, there are various built in plugins such as show, eq, ord, enum, iter, map, fold, make, yojson and protobuf.\nAnother well known plugin is ppx_fields_conv:\nGeneration of accessor and iteration functions for OCaml records. ppx_fields_conv is a ppx rewriter that can be used to define first class values representing record fields, and additional routines, to get and set record fields, iterate and fold over all fields of a record and create new record values.\nThis was the basis for the idea of Myriad. We will not be taking the full capability of ppx_fields_conv only a subset to show the potential of this approach. More specifically we will be creating field accessor functions and a create function for each record in the input.\nUsage And Demo # Input code # Myriad works with an input file as the basis for code generation, specifically records within the input file are used in the code generation phase.\nnamespace Example type Test1 = { one: int; two: string; three: float; four: float32 } type Test2 = { one: Test1; two: string } Myriad is invoked via the following addition to an F# project file:\n\u0026lt;Compile Include=\u0026#34;Generated.fs\u0026#34; \u0026gt; \u0026lt;!--1--\u0026gt; \u0026lt;MyriadFile\u0026gt;..\\..\\src\\Example\\Library.fs\u0026lt;/MyriadFile\u0026gt; \u0026lt;!--2--\u0026gt; \u0026lt;MyriadNameSpace\u0026gt;Test\u0026lt;/MyriadNameSpace\u0026gt; \u0026lt;!--3--\u0026gt; \u0026lt;/Compile\u0026gt; The \u0026lt;Compile Include=\u0026quot;...\u0026quot; element is used to specify the output name and also to make sure that the generated file is used during compilation. \u0026lt;MyriadFile\u0026gt;... is used to choose the file as input to the Myriad code generation. \u0026lt;MyriadNameSpace\u0026gt;... is used to specify a namespace to use for the generated code. If this is omitted then the RootNamespace from the project file is used. Output code # //------------------------------------------------------------------------------ // This code was generated by myriad. // Changes to this file will be lost when the code is regenerated. //------------------------------------------------------------------------------ namespace rec Test module Test1 = open Example let one (x : Test1) = x.one let two (x : Test1) = x.two let three (x : Test1) = x.three let four (x : Test1) = x.four let create (one : int) (two : string) (three : float) (four : float32) : Test1 = { one = one two = two three = three four = four } module Test2 = open Example let one (x : Test2) = x.one let two (x : Test2) = x.two let create (one : Test1) (two : string) : Test2 = { one = one two = two } The technical aspects of this project derive from four main areas: parsing, AST construction, code output and build integration.\nParsing # The first step is gathering information from the input file which is easily done using FSharp.Compiler.Services. Its easy enough to extract the AST from a piece of code using something such as:\nlet filename = \u0026#34;test.fs\u0026#34; let fileText = File.ReadAllText filename let checker = FSharpChecker.Create() let projOptions, _ = checker.GetProjectOptionsFromScript(filename, fileText) |\u0026gt; Async.RunSynchronously let ast = let parsingOptions, _ = checker.GetParsingOptionsFromProjectOptions(projOptions) let parseFileResults = checker.ParseFile(file, input, parsingOptions) |\u0026gt; Async.RunSynchronously match parseFileResults.ParseTree with | Some tree -\u0026gt; tree | None -\u0026gt; failwith \u0026#34;Something went wrong during parsing!\u0026#34; A checker is created and projectOptions are created using the filename and fileText as input. Now the checker can be used to extract an ast by first creating parsingOptions and passing them to the checker.ParseFile, this function returns an option which we pattern match, throwing an exception if there is no Ast present.\nNow that we have the Ast we can use more pattern matching to try and find Ast nodes that we are interested in. This can be done using a small section of dense pattern matching and decomposition:\nmatch ast with | ParsedInput.ImplFile(ParsedImplFileInput(_,_,_,_,_,modules,_)) -\u0026gt; for SynModuleOrNamespace(namespaceIdent,_,_,moduleDecls,_,_,_,_) in modules do for moduleDecl in moduleDecls do match moduleDecl with | SynModuleDecl.Types(types,_) -\u0026gt; for TypeDefn(ComponentInfo(_,_,_,recordIdent,_,_,_,_), typeDefRepr,_,_) in types do match typeDefRepr with | SynTypeDefnRepr.Simple(SynTypeDefnSimpleRepr.Record(_,fields,_),_) -\u0026gt; yield (namespaceIdent,recordIdent,fields) //... In this snippet you can see that we traverse the AST first decomposing ParsedInput.ImplFile, we do this so that we don\u0026rsquo;t have to extract further information in another match such as:\nmatch ast with | ParsedInput.ImplFile(pu) -\u0026gt; match pu with ParsedImplFileInput(_,_,_,_,_,modules,_) Now we can loop through the modules/namespaces. We then drill deeper until we find type definitions within the module. Once we have found a type definition we can then match on a record node which is a SynTypeDefnSimpleRepr.Record type. Once we have found a record we can extract and yield and parameters that we need for the generator. In this instance all we need is the parent namespace which we can find from SynModuleOrNamespace(namespaceIdent,_,_,_,_,_,_,_), the record identifier which we find in the type definitions ComponentInfo: TypeDefn(ComponentInfo(_,_,_,recordIdent,_,_,_,_),_,_,_). The final parameter we need is the fields of the record which are in the record definition itself: SynTypeDefnSimpleRepr.Record(_,fields,_). In this section you can see that we heavily used pattern matching and discriminated union decomposition, which are ideal for this particular task.\nAst Construction # Now that we have the information we need we can now go about constructing the modules and functions that we want to generate.\nTo help with AST construction we use a library called FSAst. I also used this in Falanx but thought I would explain it more detail here.\nThe principle behind FsAst is that it wraps common AST nodes with record types that have default values, this allows us to use record update syntax. Most of the AST nodes have a lot of parameters and it can be annoying and cumbersome to construct them.\nFor example this is an example of Let x = 42:\nLet is a node of the type SynModuleDecl, here is the definition:\nSynModuleDecl = | ModuleAbbrev of ident: Ident * longId: LongIdent * range: range | NestedModule of SynComponentInfo * isRecursive: bool * SynModuleDecls * bool * range: range | Let of isRecursive: bool * SynBinding list * range: range | DoExpr of SequencePointInfoForBinding * SynExpr * range: range | Types of SynTypeDefn list * range: range | Exception of SynExceptionDefn * range: range | Open of longDotId: LongIdentWithDots * range: range | Attributes of SynAttributes * range: range | HashDirective of ParsedHashDirective * range: range | NamespaceFragment of SynModuleOrNamespace The single Let binding Let x = 42 looks like this:\nLet (false, [Binding (None,NormalBinding,false,false,[], PreXmlDoc ((2,5),Microsoft.FSharp.Compiler.Ast+XmlDocCollector), SynValData (None,SynValInfo ([],SynArgInfo ([],false,None)),None), Named (Wild tmp.fsx (2,4--2,5) IsSynthetic=false,x,false,None, tmp.fsx (2,4--2,5) IsSynthetic=false),None, Const (Int32 42,tmp.fsx (2,8--2,10) IsSynthetic=false), tmp.fsx (2,4--2,5) IsSynthetic=false, SequencePointAtBinding tmp.fsx (2,0--2,10) IsSynthetic=false)], tmp.fsx (2,0--2,10) IsSynthetic=false) On its own it would be defined as SynModuleDecl.Let(false, bindings, range), which would require the construction of a list of Bindings, a Binding is defined like this:\nSynBinding = | Binding of accessibility: SynAccess option * kind: SynBindingKind * mustInline: bool * isMutable: bool * attrs: SynAttributes * xmlDoc: PreXmlDoc * valData: SynValData * headPat: SynPat * returnInfo: SynBindingReturnInfo option * expr: SynExpr * range: range * seqPoint: SequencePointInfoForBinding SynBinding.Binding( None, NormalBinding, false, false, [], PreXmlDoc(range.zero, XmlDocCollector()), SynValData(None, SynValInfo([], SynArgInfo ([], false, None)), None), Named(Wild, \u0026#34;tmp.fsx, range.zero, IsSynthetic=false, \u0026#34;x\u0026#34;, false, None, \u0026#34;tmp.fsx\u0026#34;, range.zero, IsSynthetic=false), None, Const (Int32, 42,\u0026#34;tmp.fsx\u0026#34;, range.zero, IsSynthetic=false), \u0026#34;tmp.fsx\u0026#34;, range.zero, IsSynthetic=false, SequencePointAtBinding(\u0026#34;tmp.fsx\u0026#34;, range.zero, IsSynthetic=false) You can see that defining these nodes can get complicated really quickly. With FsAst we can define the above Let AST fragment like this:\nSynModuleDecl.CreateLet( { SynBindingRcd.Let with Pattern = SynPatRcd.CreateNamed(Ident.Create \u0026#34;x\u0026#34;, SynPatRcd.CreateWild) Expr = SynExpr.CreateConst(SynConst.Int32 42) } ) This makes constructing aST fragments a lot easier!\nHere is a snippet of code from Myriad which create a field mapping for a record:\nlet createMap (parent: LongIdent) (field: SynField) = let field = field.ToRcd let fieldName = match field.Id with None -\u0026gt; failwith \u0026#34;no field name\u0026#34; | Some f -\u0026gt; f let recordType = LongIdentWithDots.Create (parent |\u0026gt; List.map (fun i -\u0026gt; i.idText)) |\u0026gt; SynType.CreateLongIdent let varName = \u0026#34;x\u0026#34; let pattern = let name = LongIdentWithDots.Create([fieldName.idText]) let arg = let named = SynPatRcd.CreateNamed(Ident.Create varName, SynPatRcd.CreateWild ) SynPatRcd.CreateTyped(named, recordType) |\u0026gt; SynPatRcd.CreateParen SynPatRcd.CreateLongIdent(name, [arg]) let expr = let ident = LongIdentWithDots.Create [ yield varName; yield fieldName.idText] SynExpr.CreateLongIdent(false, ident, None) let valData = let argInfo = SynArgInfo.CreateIdString \u0026#34;x\u0026#34; let valInfo = SynValInfo.SynValInfo([[argInfo]], SynArgInfo.Empty) SynValData.SynValData(None, valInfo, None) SynModuleDecl.CreateLet [{SynBindingRcd.Let with Pattern = pattern Expr = expr ValData = valData }] Let me run through the sections that make up a Let binding, lets use this an an example:\nlet one (x : Test1) = x.one Pattern # Pattern is the bindings name, in this case its: one (x : Test1) We create a LongIdentifier from the fieldNames.idText as the name of the Let binding will be the same as the field name, we then make an argument using \u0026quot;x\u0026quot; as the varName\nExpr # Expr is the expression that you are binding to the name, so this is: x.one, which is essentially just a LongIdentWithDots of varName . fieldName.\nValData # ValData Is information about the argument names and other metadata for a parameter for a member or function. Such as if the parameter optional or any attributes applied to the parameter. In this instance we just add the identifier for the argument.\nUsing FsAst makes things a lot easier to build F# AST\u0026rsquo;s with code, but there is still a lot of improvements that could be made with Ident creation and other areas, there are possibly lots of functions withing the compiler that could be exposed to help with this too.\nCode Output # Actual code generation can be done using the F# formatter tool Fantomas. Fantomas has an API call that accepts an AST and formats the code that it represents. All we have to do is make a call to that API to get back formatted source code and append it onto a header:\nlet sourceCode = Fantomas.CodeFormatter.FormatAST(ast, filename, None, fantomasConfig) This can now have a header inserted and be written to a file:\nlet code = [ \u0026#34;//------------------------------------------------------------------------------\u0026#34; \u0026#34;// This code was generated by myriad.\u0026#34; \u0026#34;// Changes to this file will be lost when the code is regenerated.\u0026#34; \u0026#34;//------------------------------------------------------------------------------\u0026#34; formattedCode ] |\u0026gt; String.concat Environment.NewLine File.WriteAllText(outputFile, code) MSBuild integration # Wrapping the parsing and generation of code in a manner that is easy to use is done via an MSBuild extension, this gives a close approximation to the use of ppx_deriving and its role within the OCaml ecosystem.\nThis is achieved by adding two child attributes to the Compile MSBuild element as follows:\n\u0026lt;Compile Include=\u0026#34;Generated.fs\u0026#34;\u0026gt; \u0026lt;MyriadFile\u0026gt;..\\..\\src\\Example\\Library.fs\u0026lt;/MyriadFile\u0026gt; \u0026lt;MyriadNameSpace\u0026gt;Test\u0026lt;/MyriadNameSpace\u0026gt; \u0026lt;/Compile\u0026gt; The \u0026lt;Compile Include=\u0026quot;Generated.fs\u0026quot; \u0026gt; element is used to specify the output name and also to make sure that the generated file is used during compilation.\n\u0026lt;MyriadFile\u0026gt;..\\..\\src\\Example\\Library.fs\u0026lt;/MyriadFile\u0026gt; is used to choose the file as input to the code generation.\n\u0026lt;MyriadNameSpace\u0026gt;Test\u0026lt;/MyriadNameSpace\u0026gt; is used to specify a namespace to use for the generated code. If this is omitted then RootNamespace is used.\n_MyriadSdkFilesList Target # In order for the integration to occur MyriadFile and MyriadNameSpace have to be processed by the MSBuild extension to form a list of Compile element extensions that we can then use to form as an input to a CLI/Command line tool. This is done in the _MyriadSdkFilesList Target, this first part is shown below:\n\u0026lt;Target Name=\u0026#34;_MyriadSdkFilesList\u0026#34; BeforeTargets=\u0026#34;MyriadSdkGenerateInputCache\u0026#34;\u0026gt; \u0026lt;ItemGroup\u0026gt; \u0026lt;MyriadSource Include=\u0026#34;%(Compile.MyriadFile)\u0026#34; Condition=\u0026#34; \u0026#39;%(Compile.MyriadFile)\u0026#39; != \u0026#39;\u0026#39; \u0026#34;\u0026gt; \u0026lt;OutputPath\u0026gt;$([System.IO.Path]::GetFullPath(\u0026#39;%(Compile.FullPath)\u0026#39;))\u0026lt;/OutputPath\u0026gt; \u0026lt;Namespace Condition=\u0026#34; \u0026#39;%(Compile.MyriadNamespace)\u0026#39; != \u0026#39;\u0026#39; \u0026#34; \u0026gt;%(Compile.MyriadNamespace)\u0026lt;/Namespace\u0026gt; \u0026lt;Namespace Condition=\u0026#34; \u0026#39;%(Compile.MyriadNamespace)\u0026#39; == \u0026#39;\u0026#39; \u0026#34; \u0026gt;$(RootNamespace)\u0026lt;/Namespace\u0026gt; \u0026lt;/MyriadSource\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;ItemGroup\u0026gt; \u0026lt;MyriadCodegen Include=\u0026#34;%(MyriadSource.FullPath)\u0026#34;\u0026gt; \u0026lt;OutputPath Condition=\u0026#34; \u0026#39;%(MyriadSource.OutputPath)\u0026#39; != \u0026#39;\u0026#39; \u0026#34;\u0026gt;$([System.IO.Path]::GetFullPath(\u0026#39;%(MyriadSource.OutputPath)\u0026#39;))\u0026lt;/OutputPath\u0026gt; \u0026lt;OutputPath Condition=\u0026#34; \u0026#39;%(MyriadSource.OutputPath)\u0026#39; == \u0026#39;\u0026#39; \u0026#34;\u0026gt;%(MyriadSource.FullPath).fs\u0026lt;/OutputPath\u0026gt; \u0026lt;Namespace\u0026gt;%(MyriadSource.Namespace)\u0026lt;/Namespace\u0026gt; \u0026lt;/MyriadCodegen\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;PropertyGroup\u0026gt; \u0026lt;_MyriadSdkCodeGenInputCache\u0026gt;$(IntermediateOutputPath)$(MSBuildProjectFile).FalanxSdkCodeGenInputs.cache\u0026lt;/_MyriadSdkCodeGenInputCache\u0026gt; \u0026lt;/PropertyGroup\u0026gt; \u0026lt;/Target\u0026gt; We first gather a list of files for Myriad to process. We do this by creating an ItemGroup which is a list of MyriadSource elements, only Compile elements that have a MyriadFile node are processed, this is done via the Condition attribute: Condition=\u0026quot; '%(Compile.MyriadFile)' != ''. The MyriadSource element is formed from three pieces of information.\nThe Include attribute is the MyriadFile element we include in the MSBuild file.\nThe OutputPath element is full path for the Compile elements Include attribute, this is also known as Identity The Namespace element is either the %(Compile.MyriadNamespace) if it is present or the $(RootNamespace) if it is not.\nNow that we have created an ItemGroup containing MyriadSource elements we can refine this a little, you could fold these changes into the MyriadSource ItemGroup but it is easier to create two ItemGroup elements.\nWe create a new ItemGroup called MyriadCodegen which references MyriadSource for its Include attribute. There are also Condition attributes to check the OutputPath is not empty and also another to ensure that if it is empty to just set it to the input file. This would mean that the input file itself would be processed and changed rather than being written to another file.\nMyriadSdkGenerateCode Target # The final step is to invoke the CLI tool with all the information we have gathered in the MyriadSdkGenerateCode target:\n\u0026lt;PropertyGroup\u0026gt; \u0026lt;MyriadSdkGenerateCodeDependsOn\u0026gt;$(MyriadSdkGenerateCodeDependsOn);ResolveReferences;MyriadSdkGenerateInputCache\u0026lt;/MyriadSdkGenerateCodeDependsOn\u0026gt; \u0026lt;/PropertyGroup\u0026gt; \u0026lt;Target Name=\u0026#34;MyriadSdkGenerateCode\u0026#34; DependsOnTargets=\u0026#34;$(MyriadSdkGenerateCodeDependsOn)\u0026#34; BeforeTargets=\u0026#34;CoreCompile\u0026#34; Condition=\u0026#34; \u0026#39;$(DesignTimeBuild)\u0026#39; != \u0026#39;true\u0026#39; \u0026#34; Inputs=\u0026#34;@(MyriadCodegen);$(_MyriadSdkCodeGenInputCache);$(MyriadSdk_Generator_Exe)\u0026#34; Outputs=\u0026#34;%(MyriadCodegen.OutputPath)\u0026#34;\u0026gt; \u0026lt;PropertyGroup\u0026gt; \u0026lt;_MyriadSdk_InputFileName\u0026gt;%(MyriadCodegen.Identity)\u0026lt;/_MyriadSdk_InputFileName\u0026gt; \u0026lt;_MyriadSdk_OutputFileName\u0026gt;%(MyriadCodegen.OutputPath)\u0026lt;/_MyriadSdk_OutputFileName\u0026gt; \u0026lt;_MyriadSdk_Namespace\u0026gt;%(MyriadCodegen.Namespace)\u0026lt;/_MyriadSdk_Namespace\u0026gt; \u0026lt;/PropertyGroup\u0026gt; \u0026lt;ItemGroup\u0026gt; \u0026lt;MyriadSdk_Args Include=\u0026#39;--inputfile \u0026#34;$(_MyriadSdk_InputFileName)\u0026#34;\u0026#39; /\u0026gt; \u0026lt;MyriadSdk_Args Include=\u0026#39;--outputfile \u0026#34;$(_MyriadSdk_OutputFileName)\u0026#34;\u0026#39; /\u0026gt; \u0026lt;MyriadSdk_Args Include=\u0026#39;--namespace \u0026#34;$(_MyriadSdk_Namespace)\u0026#34;\u0026#39; /\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;!-- Use dotnet to execute the process. --\u0026gt; \u0026lt;Exec Command=\u0026#34;$(MyriadSdk_Generator_ExeHost)\u0026amp;quot;$(MyriadSdk_Generator_Exe)\u0026amp;quot; @(MyriadSdk_Args -\u0026gt; \u0026#39;%(Identity)\u0026#39;, \u0026#39; \u0026#39;)\u0026#34; /\u0026gt; \u0026lt;/Target\u0026gt; The DependsOnTargets attribute is used to ensure that anything contained in the MyriadSdkGenerateCodeDependsOn element is ran before this Target.\nWithin the MyriadSdkGenerateCode Target element there are Inputs and Outputs attributes, these are used to determine when Myriad needs to run. An item is considered up-to-date if its output file is the same age or newer than its input file or files.\nWe create a PropertyGroup to contain the command line parameters _MyriadSdk_InputFileName, _MyriadSdk_OutputFileName and _MyriadSdk_Namespace using the corresponding elements from the _MyriadSdkFilesList Targets ItemGroup MyriadCodegen.\nwe now create an ItemGroup which has within it three MyriadSdk_Args elements that we need to invoke the code generator with.\nFinally we execute the code generator with Exec invoking the CLI too, with the parameters from MyriadSdk_Args Include attribute via the MSBuild function @(MyriadSdk_Args -\u0026gt; '%(Identity)', ' ')\nOne thing that was not discussed in this section was the _MyriadSdkCodeGenInputCache that was referenced in both targets above:\n\u0026lt;Target Name=\u0026#34;MyriadSdkGenerateInputCache\u0026#34; DependsOnTargets=\u0026#34;ResolveAssemblyReferences;_MyriadSdkFilesList\u0026#34; BeforeTargets=\u0026#34;MyriadSdkGenerateCode\u0026#34;\u0026gt; \u0026lt;ItemGroup\u0026gt; \u0026lt;MyriadSdk_CodeGenInputs Include=\u0026#34;@(MyriadCodegen);@(ReferencePath);$(MyriadSdk_Generator_Exe)\u0026#34; /\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;Hash ItemsToHash=\u0026#34;@(MyriadSdk_CodeGenInputs)\u0026#34;\u0026gt; \u0026lt;Output TaskParameter=\u0026#34;HashResult\u0026#34; PropertyName=\u0026#34;MyriadSdk_UpdatedInputCacheContents\u0026#34; /\u0026gt; \u0026lt;/Hash\u0026gt; \u0026lt;WriteLinesToFile Overwrite=\u0026#34;true\u0026#34; File=\u0026#34;$(_MyriadSdkCodeGenInputCache)\u0026#34; Lines=\u0026#34;$(MyriadSdk_UpdatedInputCacheContents)\u0026#34; WriteOnlyWhenDifferent=\u0026#34;True\u0026#34; /\u0026gt; \u0026lt;/Target\u0026gt; This target generates a hash using @(MyriadCodegen);@(ReferencePath);$(MyriadSdk_Generator_Exe) as an input. If any of those changes then a different hash will be written to the output file via \u0026lt;WriteLinesToFile Overwrite=\u0026quot;true\u0026quot; File=\u0026quot;$(_MyriadSdkCodeGenInputCache)\u0026quot;. This captures the total set of all inputs to the code generator. This is based on the _GenerateCompileDependencyCache target from the .NET project system, which was used as a reference. You can find this in the .Net project system source.\nSummary # Wow this was a lot longer than I thought it would be, I hope I didn\u0026rsquo;t ramble too much, its quite a large area once you try to dissect the details. This are probably multiple essays I could write on each topic.\nI think the key points I summed up at the end of Falanx were that quotations were not easy to work with, and thats certainly true if you try pushing them right to their limits. I think Type Providers and applicative programming with many generic parameters certainly does that!\nFalanx was a great project that showed that even out of the box libraries can be used to compose quite a complex project. Building up a quotations from both literals and expressions takes a lot of time and often you have to run a lot of debug cycles to get things right. Myriad built on that by providing a simpler solution of composing the AST directly. I think there is some scope for a middle ground between the two where a quotation literals could be used as shortcut to provided an AST snippet much in the same way you can do this with Haxe Expression Reification. Its sort of similar to the way you can use quotations to extract a MethodInfo. This is used quote a bit in Falanx.Machinery:\n\u0026lt;@@ writeEmbedded x x x @@\u0026gt; |\u0026gt; Expr.methodof |\u0026gt; Expr.callStatic [position; buffer; value] I think some creative uses of moving to and from quotations and the AST would allow a lot more flexibility in meta-programming.\nYou can find the repositories for both Myriad and Falanx on github as soon as Ive made this post public, they are both also available on Nuget too: Falanx.Sdk/Myriad.Sdk\nI hope you enjoyed this post on applied meta-programming and hope to expand on this subject and many others on my YouTube channel.\nUntil next time!\nReferences # Type-driven code generation for OCaml - ppx_deriving\nGeneration of accessor and iteration functions for OCaml records\nCompiler Services: Processing untyped syntax tree\nFantomas F# source code formatter\n.NET project System:- _GenerateCompileDependencyCache\nFalanx protobuf Code Generation\nMeta-matic\nType Provider SDK\nType Providers\nProtocol Buffers\nFleece\nFroto\nQuotation Compiler\nStatically Resolved Type Parameters\nQuotations\nFsAst\nMyriad\nMy YouTube Channel\nHaxe\nExpression Reification\nCLI Tool\nDiscriminated Unions\nRecords\n","date":"April 24, 2019","externalUrl":null,"permalink":"/programming/2019-04-24-applied-metaprogramming-with-myriad/","section":"Blog","summary":"This article is all about F# metaprogramming and research that I did which evolved over time, firstly with a project called Falanx that I developed while working for Jet/Walmart, then later the ideas and concepts evolved into Myriad.\n","title":"Applied Meta-Programming with Myriad","type":"programming"},{"content":"","date":"April 24, 2019","externalUrl":null,"permalink":"/tags/typeproviders/","section":"Tags","summary":"","title":"Typeproviders","type":"tags"},{"content":"Hi, its been a very long while since I last blogged, I promised this ages ago, its a partial sample chapter for a book I was planning on Elm but the publisher started to muck me about so it never happened.\nI also have the table of contents for the rest of the book but I guess your not interested in a book Im not writing so I have omitted that. Anyway heres the sample chapter in its raw form.\nBuilding a single page application # In this chapter we will specify and build a single page application consisting of bootstrap based navbar navigation with a selection of views that are shown based on the navigation option selected within the applications model.\nBasic Structure # The basic structure of the application we will be constructing comprises a home, about, contact, gallery and item view as follows:\nHome View # This will contain a basic navigation bar at the top with the contents reflecting the current navigation selection:\nNavigation bar: [About|Gallery|Contact] \u0026lt;Main content view, this is based on the current navigation selection\u0026gt; \u0026lt;fixed footer\u0026gt; About View # The About view is simply a page displaying details, like a description and image.\n\u0026lt;text\u0026gt; \u0026lt;image\u0026gt; \u0026lt;back\u0026gt; Gallery View # The gallery view is a list of categories showing an image and description for each. Clicking on the image result in navigating to the Item View, theres also a back button to navigate back to Home.\n\u0026lt;fixed description text\u0026gt; Category1 image Category2 image Category3 image \u0026lt;back button\u0026gt; Contact View # The contact view contain some text and links to various social media, and a link to navigate back to Home.\n\u0026lt;text\u0026gt; \u0026lt;social media link1\u0026gt; \u0026lt;social media link2\u0026gt; \u0026lt;back button\u0026gt; Item View # The item view contains a image and descriptive text as well as a means to navigate back to the Gallery.\n\u0026lt;text\u0026gt; \u0026lt;image\u0026gt; \u0026lt;back button\u0026gt; The basic structure of this single page application is relatively simple and also follows The elm architecture as you would expect.\nThis single page application will be split into files loosely based on How i structure elm apps by Kris Jenkins.\n├─ App.elm ├─ State.elm ├─ Types.elm ├─ View.elm The main application startup will be hosted in App.elm. Application state and models will be contained within State.elm. Types.elm will contain the various types that we will be using in the application. Finally the initial view visible to the user will be contained in View.elm.\nIn addition each view in the application can be given its own module and file nested in the file structure as follows:\n├─ About │ └─ View.elm ├─ Category │ └─ View.elm ├─ Contact │ └─ View.elm ├─ Detail │ └─ View.elm ├─ Gallery │ └─ View.elm └─ Home └─ View.elm Although this single page application is relatively simple and could be built by any one of the many static site engines like Ghost, Hugo or Jekyll it builds on the earlier chapters slowly adding complexity so you can see where things would lead to on a bigger site more complex site where you have additional requirements like web sockets etc.\nCore structure # First of all lets look at how we can construct the main entry point of the application, let’s create the following skeleton for App.elm:\nmodule Main exposing (..) type alias Model = { page : string } init = { page = \u0026#34;Home\u0026#34; } view model = div [] [] update msg model = (model, Cmd.none) subscription model = Sub.None main : Program Never main = Html.program { init = init , view = view , update = update , subscriptions = subscriptions } All the main elements of the application are imported and a Html.program is started with the appropriate init, view, update, and subscriptions.\nLets start to flesh things out further by further defining the model and other related types.\nModel, Messages and types # In this section we will be describing the model which represents the applications state as runtime, the messages used within the application and also any types that we might need to represent in the application too.\nModel # At the moment the model is really simple, its just a record with a single field page which is a string. Thats not a very robust way of defining the model so lets see about changing that now.\nThe current page of the application is fairly well defined its either going to be one of the following pages:\nHome - the default home page About - a page which displays information about the site Contact- a page which displays contact details after first exposing a captcha request Gallery - a page which shows the user different categories that are available to view Gallery Category - a page which shows a list of items available in a a category Item Detail - a page showing details on a single item We can model the concept of a page really well with a union type:\ntype Page = Home | About | Contact | Gallery | CategoryDetail String | ItemDetail String Home, About, Contact and Gallery are depicted by empty types with no specific shape but CategoryDetail and ItemDetail have the shape or type of a string. Come to think of it CategoryDetail is very loosely typed being represented by a simple string, we can tighten that up too as the set of categories is also well defined. Lets just define just three categories for now:\ntype CategoryDetail = Seasides | IllustratedQuotes | Architecture``` The same cannot be said about ItemDetail as thats just going to be a key to the items name, it could equally be a number but lets keep that as a simple string for now. Lets update the Page type to take those new types into account:\ntype Page = Home | About | Contact | Gallery | CategoryDetail CategoryDetail | ItemDetail String We can now also update the Model so that the field page is represented by the Page type:\ntype alias Model = { page : Page } Messages # The primary messages that will be used will be either no navigate back a page or to navigate to a specific page. There will also be a message to indicate we want to use a captcha to view an email address to avoid an email address being exposed to crawlers. Lets define these three messages again using union types:\ntype Msg = NavigateTo Page | NavigateBack | Captcha NavigateTo - navigates to the page detailed. NavigateBack - navigates back one page. Captcha - displays a captcha request which then shows an email address on success. More types # In the application we will also need types to represent the information about the categories detail and Item detail, we can use records to do this with simple string fields to represent textual information and images. Category will have a categoryType, img and description:\ntype alias Category = {categoryType : CategoryType, img : String, description : String} Item will have an id, title, img, description and category:\ntype alias Item = {id : String, title : String, img : String, description : String, category : CategoryType} All of the field types are simple types apart from category which is also a CategoryType which we defined above.\nView # The View is quite simple based on a standard Bootstrap 4 Navbar navigation. We can define it by adapting some standard bootstrap html:\n\u0026lt;nav class=\u0026#34;navbar navbar-light bg-faded\u0026#34;\u0026gt; \u0026lt;a class=\u0026#34;navbar-brand\u0026#34; href=\u0026#34;#\u0026#34;\u0026gt;Navbar\u0026lt;/a\u0026gt; \u0026lt;ul class=\u0026#34;nav navbar-nav\u0026#34;\u0026gt; \u0026lt;li class=\u0026#34;nav-item active\u0026#34;\u0026gt; \u0026lt;a class=\u0026#34;nav-link\u0026#34; href=\u0026#34;#\u0026#34;\u0026gt;Home \u0026lt;span class=\u0026#34;sr-only\u0026#34;\u0026gt;(current)\u0026lt;/span\u0026gt;\u0026lt;/a\u0026gt; \u0026lt;/li\u0026gt; \u0026lt;li class=\u0026#34;nav-item\u0026#34;\u0026gt; \u0026lt;a class=\u0026#34;nav-link\u0026#34; href=\u0026#34;#\u0026#34;\u0026gt;Link\u0026lt;/a\u0026gt; \u0026lt;/li\u0026gt; \u0026lt;li class=\u0026#34;nav-item\u0026#34;\u0026gt; \u0026lt;a class=\u0026#34;nav-link\u0026#34; href=\u0026#34;#\u0026#34;\u0026gt;Link\u0026lt;/a\u0026gt; \u0026lt;/li\u0026gt; \u0026lt;li class=\u0026#34;nav-item dropdown\u0026#34;\u0026gt; \u0026lt;a class=\u0026#34;nav-link dropdown-toggle\u0026#34; href=\u0026#34;http://example.com\u0026#34; id=\u0026#34;supportedContentDropdown\u0026#34; data-toggle=\u0026#34;dropdown\u0026#34; aria-haspopup=\u0026#34;true\u0026#34; aria-expanded=\u0026#34;false\u0026#34;\u0026gt;Dropdown\u0026lt;/a\u0026gt; \u0026lt;div class=\u0026#34;dropdown-menu\u0026#34; aria-labelledby=\u0026#34;supportedContentDropdown\u0026#34;\u0026gt; \u0026lt;a class=\u0026#34;dropdown-item\u0026#34; href=\u0026#34;#\u0026#34;\u0026gt;Action\u0026lt;/a\u0026gt; \u0026lt;a class=\u0026#34;dropdown-item\u0026#34; href=\u0026#34;#\u0026#34;\u0026gt;Another action\u0026lt;/a\u0026gt; \u0026lt;a class=\u0026#34;dropdown-item\u0026#34; href=\u0026#34;#\u0026#34;\u0026gt;Something else here\u0026lt;/a\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/li\u0026gt; \u0026lt;/ul\u0026gt; \u0026lt;/nav\u0026gt; We can define the following rootView function:\nrootView model = div [ class \u0026#34;container\u0026#34; ] [ nav [ class \u0026#34;navbar navbar-light\u0026#34;, attribute \u0026#34;role\u0026#34; \u0026#34;navigation\u0026#34; ] [ a [ class \u0026#34;pull-xs-left\u0026#34; , href \u0026lt;| toHash Home , onClick_ \u0026lt;| NavigateTo Home ] [ img [ id \u0026#34;logo\u0026#34;, class \u0026#34;img-fluid\u0026#34;, src \u0026#34;/img/logogreen.png\u0026#34;, srcset [ \u0026#34;/img/logogreen.png\u0026#34;, \u0026#34;/img/logogreen@2x.png\u0026#34; ] ] [] ] , button [ attribute \u0026#34;aria-controls\u0026#34; \u0026#34;exCollapsingNavbar2\u0026#34; , attribute \u0026#34;aria-expanded\u0026#34; \u0026#34;false\u0026#34; , attribute \u0026#34;aria-label\u0026#34; \u0026#34;Toggle navigation\u0026#34; , class \u0026#34;navbar-toggler hidden-sm-up flex-center\u0026#34; , attribute \u0026#34;data-target\u0026#34; \u0026#34;#exCollapsingNavbar2\u0026#34; , attribute \u0026#34;data-toggle\u0026#34; \u0026#34;collapse\u0026#34; , type\u0026#39; \u0026#34;button\u0026#34; ] [ text \u0026#34;☰\u0026#34; ] , div [ class \u0026#34;collapse navbar-toggleable-xs\u0026#34;, id \u0026#34;exCollapsingNavbar2\u0026#34; ] [ ul [ class \u0026#34;nav navbar-nav pull-sm-right text-xs-center\u0026#34; ] [ renderMenuItem model Home \u0026#34;Home\u0026#34; , renderMenuItem model About \u0026#34;About\u0026#34; , renderMenuItem model Gallery \u0026#34;Gallery\u0026#34; , renderMenuItem model Contact \u0026#34;Contact\u0026#34; ] ] ] , div [ class \u0026#34;content container-fluid\u0026#34; ] [ viewPage model ] ] Theres are several functions here which we have not seen before: toHref, onClick_, srcset, renderMenuItem and viewPage, lets go though them now.\nConverting pages to hrefs # When we want to navigate we can either use a href node or send a command to navigate via the Elm architecture. If we want to navigate using href then we need some way to convert our representation of a page into something that can be represented in the url in the browser navigation:\ntoHref : Page -\u0026gt; String toHash page = 1case page of Home -\u0026gt; 2\u0026#34;/home\u0026#34; About -\u0026gt; \u0026#34;/about\u0026#34; Contact -\u0026gt; \u0026#34;/contact\u0026#34; Gallery -\u0026gt; \u0026#34;/gallery\u0026#34; CategoryDetail categoryType -\u0026gt; 3\u0026#34;/category/\u0026#34; ++ toString categoryType ItemDetail name -\u0026gt; \u0026#34;/item/\u0026#34; ++ name The case statement is used to start pattern matching on the page type, we will have a separate statement to handle each case in the page union type. For a simple page like About, we return a string that represents a simple path. For a more complex page like CategoryDetail we have to combine elements from the union type to build a path. Here you can see that the categoryType is concatenated not the string \u0026ldquo;category/\u0026rdquo;. The function toString is used to convert categoryType into a string as it is also a union type.\nonClick event handling # The astute reader may have noticed the onClick_ with a trailing underscore which is different to the normal definition of onClick. The onClick event is also working hand in hand with the href to allow us to navigate with our own event handler. We define a slightly different onClick as follows:\nonClick_ : a -\u0026gt; Attribute a onClick_ msg = onWithOptions \u0026#34;click\u0026#34; { stopPropagation = True, preventDefault = True} (succeed msg) Whats happening here is we are using onWithOptions to define an alternative onClick thats stops the propagation of the event and also overrides the default browser behaviour. If we did not do this then clicking on the link would result in the browser trying to open the href which would result in a page not found 404 error as the page does not actually exist, rather it is generated by the Elm architecture during the navigation and updating of the application.\nMultiple images sources based on pixel density # srcset is an html 5 img attribute that can be used to apply different images depending on the native resolution of the browser, [Mozilla.org][6] describes it as follows:\nA list of one or more strings separated by commas indicating a set of possible image sources for the user agent to use. Each string is composed of: a URL to an image, optionally, whitespace followed by one of: a width descriptor, or a positive integer directly followed by \u0026lsquo;w\u0026rsquo;. The width descriptor is divided by the source size given in the sizes attribute to calculate the effective pixel density. a pixel density descriptor, which is a positive floating point number directly followed by \u0026lsquo;x\u0026rsquo;.\nWe can define srcset is defined as follows:\nsrcset : List String -\u0026gt; Attribute a srcset items = let maps = 1items |\u0026gt; List.indexedMap (\\i item -\u0026gt; item ++ \u0026#34; \u0026#34; ++ toString (i + 1) ++ \u0026#34;x\u0026#34;) in property \u0026#34;srcset\u0026#34; (2maps |\u0026gt; String.join \u0026#34;,\u0026#34; \u0026gt;\u0026gt; Json.Encode.string) An indexed map is applied to each of the strings in items, we use the index to define the pixel density descriptor. The mapped strings are joined back together with String.join and encoded as a Json string. So in the view where we see:\nsrcset [\u0026#34;/img/logogreen.png\u0026#34;,\u0026#34;/img/logogreen@2x.png\u0026#34; ] we will get the following attribute:\n\u0026lt;img srcset=\u0026#34;/img/logogreen.png 1x,/img/logogreen@2x.png 2x”\u0026gt; Defining navigation and changing the appearance # renderMenuItem is changing a navigations item\u0026rsquo;s style based on the current page and also defines the navigation to a specific page:\nrenderMenuItem : Model -\u0026gt; Page -\u0026gt; String -\u0026gt; Html.Html Msg renderMenuItem model navigationPage txt = let liClass = //1 (if model.page == navigationPage then \u0026#34;nav-item active\u0026#34; else \u0026#34;nav-item\u0026#34;) textElement = //2 if model.page == menuItem then [ text txt , span [ class \u0026#34;sr-only\u0026#34; ] [ text \u0026#34;(current)\u0026#34; ] ] else [text txt] in li [class liClass] [ a [ class \u0026#34;nav-link specialEliteFont\u0026#34; , href (toHref navigationPage), onClick_ \u0026lt;| NavigateTo navigationPage //3 ] textElement ] Here we define liClass to be either nav-item active if the current page is equal to menuItem\nif the current page is equal to menuItem We define a textElement which will have it an extra span with the class sr-only (Screen Reader only) defined and the text \u0026ldquo;(Current)\u0026rdquo;. If it is not the current page then we just use the text. This is so that screen readers will have an indication of what navigation option is active as an accessibility aid.\nNotice the onClick_ event we defined along with the NavigateTo message we defined the the Messages section.\nRendering the sub views # The sub view is shown underneath the navigation menu:\n______________ | Navigation | |____________| | | | sub view | |____________| We render the subview with viewPage, the view is updated depending on which page is current:\nviewPage model = case model.page of Home -\u0026gt; getHomePage () About -\u0026gt; getAboutPage () Gallery -\u0026gt; getGalleryAsCards () Contact -\u0026gt; getContactPage () CategoryDetail category -\u0026gt; getCategoryPageCards category ItemDetail item -\u0026gt; getItemPage item You can see there is a separate view for each page which in return a list of nodes for that particular view.\nFor each of these sub views we can create a separate module and import the function into View.elm.\nAny of the parameterless pages could be defined very simply, heres an example of what Home.View could look like:\nmodule Home.View exposing (..) import Html exposing (br, div, Html, img, p, text) import Html.Attributes exposing (class, src) getAboutPage : () -\u0026gt; Html Msg getAboutPage () = div [ class \u0026#34;container-fluid\u0026#34; ] [ text “About\u0026#34; ] We create that file in the Home folder and name the file View.Elm. Remember from Chapter 3 than Elm enforces naming of modules and file names to coincide with the name of the file and module name. So we have to ensure we have a file named View.elm in a folder named Home. The module name also has to be Home.View, don\u0026rsquo;t worry the elm compiler will call you out if you get anything wrong.\nWe can import the module and function into View.elm by adding an import statement to the top of the file:\nimport Home.View exposing (getHomePage) Heres a slightly more advanced example using one of the pages CategoryDetail with parameters Category.View:\nmodule Category.View exposing (getCategoryPageCards) import Html exposing (a, br, div, Html, img, p, text) import Html.Attributes exposing (alt, class, href, name, src) getCategoryPageCards category = let items = List.filter (\\c -\u0026gt; c.category == category) Data.items colClass = case List.length items of 1 -\u0026gt; \u0026#34;col-xs-12\u0026#34; 2 -\u0026gt; \u0026#34;col-xs-12 col-sm-6\u0026#34; _ -\u0026gt; \u0026#34;col-xs-12 col-sm-6 col-md-4\u0026#34; itemMapper item = div [ class colClass ] [ div [ class \u0026#34;card\u0026#34;] [ a [ noContextMenu , href (toHref \u0026lt;| ItemDetail item.id) , onClick_ (NavigateTo \u0026lt;| ItemDetail item.id) ] [ img [ noContextMenu, class \u0026#34;card-img-top img-fluid\u0026#34;, src item.img ] [] ] , div [ class \u0026#34;card-block\u0026#34; ] [ p [ class \u0026#34;card-text\u0026#34; ] [ text item.title ] ] ] ] in div [ class \u0026#34;container-fluid\u0026#34; ] [ div [ class \u0026#34;row\u0026#34; ] (items |\u0026gt; List.map itemMapper) , div [] [ backButton ] ] The nodes returned from getCategoryPageCards are returned as part of viewPage. Data.items are filtered by the current category and mapped into divs with onClick_ navigation to the ItemDetail page.\nUpdate # The purpose of the update function is to update our model in relation to external events, in this instance its going to be mainly navigation oriented so the update is rather simple.\nThe update function is defined as follows:\nupdate : Msg -\u0026gt; Model -\u0026gt; ( Model, Cmd b ) update msg model = case msg of 1NavigateTo page -\u0026gt; ( model, (Navigation.newUrl \u0026lt;| pageToString page) ) 2NavigateBack -\u0026gt; model =\u0026gt; (Navigation.back 1) 3Captcha -\u0026gt; ( model, captcha() ) We pattern match on the msg parameter and use the commands described in the Messages section.\nFor the NavigateTo command we update the model to the new page. For navigating back we use a command from the Navigation package to navigate back exactly one page. For the Captcha command we run another function to do that work for us, the captcha function.\nThe captcha function is defined as Port defined like this:\nport captcha : () -\u0026gt; Cmd msg We then need to add a little JavaScript subscription to the port which opens a new url with the captcha verification :\napp.ports.captcha.subscribe(function () { window.open(\u0026#39;http://www.google.com/recaptcha/mailhide/...\u0026#39;, \u0026#39;\u0026#39;, \u0026#39;toolbar=0,scrollbars=0,location=0,statusbar=0,menubar=0,resizable=0,width=500,height=300\u0026#39;); return false; }); Whats happening here is that when the captcha message is received the JavaScript subscriber is notified and a a window is opened allowing the email address to be retrieved if the captcha is successful.\nNavigation and Uri parsing # Navigation or routing as it is sometimes called in single page applications is the managing of the browser address bar without creating new requests to servers etc. This is done internally via Html 5 push state.\nNavigation in Elm is handled by the Elm Navigation package. This package provides an alternative program Navigation.program which means the one we defined in Core structure now needs to be altered. The program function in Elm Navigation has been extended with an additional two extra arguments. The main program entry point needs to be modified to look like this:\nmain : Program Never main = Navigation.program (Navigation.makeParser pathParser) { init = nit , view = view , update = update , urlUpdate = urlUpdate , subscriptions = subscriptions } The first additional argument is a Parser, there is a utility function called makeParser in the Navigation package that allows us to define a function to turn a browser Location into whatever data we want to:\nmakeParser : (Location -\u0026gt; a) -\u0026gt; Parser a In the Navigation.program above you can see this used along with the pathParser function below to parse a Location into a Page:\nNavigation.program` (Navigation.makeParser pathParser) ## Parsing\nParsing is handled with a parser combinator library defined in the url-parser package. There are other parser combinator libraries which add more functionality to url-parser but it has everything you need for most situations. We covered combinators in chapter 7 Interoperability and this is very much the same concept of combining small functions to create more complex functions and behaviour. We will only be using a small selection of combinators to parse the results:\noneOf is a combinator that will try to match one of the parsers in a list. s is a string combinator matching a particular string like \u0026ldquo;home\u0026rdquo;, \u0026ldquo;shop\u0026rdquo; etc. \u0026lt;/\u0026gt; is a combinator that matches a / character in the location like item/myitem. UrlParser.string matches any string. format Is a combinator that allows you to customise or map another Parser, here it is used to Parsed output into the union types that represent them. pathParser : Navigation.Location -\u0026gt; Result String Page pathParser location = parse identity pageParser (String.dropLeft 1 location.pathname) The pathParser functions first parameter is a function to map the successful parsing to another type here we are using the identity function to leave the result as is. The String.dropLeft 1 function is removing the leading character from location.pathname which is the leading forward slash.\npageParser : UrlParser.Parser (Page -\u0026gt; a) a pageParser = oneOf [1format Home (oneOf [ s \u0026#34;home\u0026#34;, s \u0026#34;\u0026#34; ]) ,2 format About (s \u0026#34;about\u0026#34;) , format Shop (s \u0026#34;shop\u0026#34;) , format Gallery (s \u0026#34;gallery\u0026#34;) , format Contact (s \u0026#34;contact\u0026#34;) ,3 format (stringToCategoryType \u0026gt;\u0026gt; CategoryDetail) (s \u0026#34;category\u0026#34; \u0026lt;/\u0026gt; UrlParser.string) , format ItemDetail (s \u0026#34;item\u0026#34; \u0026lt;/\u0026gt; UrlParser.string) ] The format function is part of the navigation package and is simply a function to map the parsing result in if successful, in this instance we are using the Page union type constructors to perform that map. The combinators oneOf is used to choose between several in the preceding list, in this instance s is used to match the strings home and an empty string.\nAgain format is used to construct a map to the About Page. The s combinator is again use to match the string “about” This time format is used with an extra function stringToCategoryType. As shown below, . The combinators used here are the s combinator to match the string “category”, the \u0026lt;/\u0026gt; the forward slash combinator and finally UrlParser.string which matches any string. e,g, category/Seasides\nstringToCategoryType : String -\u0026gt; CategoryType stringToCategoryType category = case category of \u0026#34;Seasides\u0026#34; -\u0026gt; Seasides \u0026#34;IllustratedQuotes\u0026#34; -\u0026gt; IllustratedQuotes _ -\u0026gt; Unknown To match the category in the url back to a CategoryType we match the corresponding string representation back into a CategoryType. Finally now that we know how parsing work we can look at the urlUpdate to see how it works:\nurlUpdate : Result a Page -\u0026gt; Model -\u0026gt; ( Model, Cmd c ) urlUpdate result model = case result of Err _ -\u0026gt; ( model, Navigation.modifyUrl (pageToString model.page) ) Ok page -\u0026gt; { model | page = page } =\u0026gt; updateAnalytics (pageToString page) We pattern match on the result witch is a Result type and if its the Ok case then we update the models page to the one passed in. If the result is an error (Err) then we modify the url with Navigation.modifyUrl just pointing it back to the previous page. pageToString simply turns the Page type back into a string. I\u0026rsquo;m going to strategically ignore updateAnalytics for now as this will be covered in the next section.\nGoogle Analytics integration # Google analytics can be easily added to any web application be simply creating an account and including the following JavaScript in your html and replacing UA-12345678-1 with your own id:\n\u0026lt;script\u0026gt; (function(i,s,o,g,r,a,m){i[\u0026#39;GoogleAnalyticsObject\u0026#39;]=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,\u0026#39;script\u0026#39;,\u0026#39;https://www.google-analytics.com/analytics.js\u0026#39;,\u0026#39;ga\u0026#39;); ga(\u0026#39;create\u0026#39;, \u0026#39;UA-12345678-1\u0026#39;, \u0026#39;auto\u0026#39;); ga(\u0026#39;require\u0026#39;, \u0026#39;linkid\u0026#39;); ga(\u0026#39;send\u0026#39;, \u0026#39;pageview\u0026#39;); \u0026lt;/script\u0026gt; The problem with this solution is that theres only one real page in the application, it would be really nice if the navigation in this application could show correctly. We need to provide a way to update Google analytics whenever the page navigation changes. Luckily Google provides a way to do this via the ga('set', 'page', page) and ga('send', ‘pageview') JavaScript functions. We can do this by defining another port. Remember the updateAnalytics function from urlUpdate above?\n{ model | page = page } =\u0026gt; updateAnalytics (pageToString page) Well thats the port we are going to define now, it looks like this:\nport updateAnalytics: String -\u0026gt; Cmd msg Now all we need to do is wire up a little more JavaScript so that when the updateAnalytics function is called the JavaScript subscriber is notified and Google analytics is updated correctly.\nThe JavaScript looks like this:\napp.ports.updateAnalytics.subscribe(function (page) { ga(\u0026#39;set\u0026#39;, \u0026#39;page\u0026#39;, page); ga(\u0026#39;send\u0026#39;, \u0026#39;pageview\u0026#39;); }); Summary # We have covered quite a range of different aspects in this chapter:\nSetting up a project with a structure that supports a lot more expansion rather than having thousands of lines crammed into a single file. We used a model and type based approach to model the navigation and pages with the application We learned how Navigation works in single page applications Used Html 5 img srcset attribute to apply different images based on pixel density Added custom events to override default browser behavior on navigation to a url Learned how to use parser combinators to parse url fragments Used ports to communicate with JavaScript Solved a problem with the correct navigation been shown in single page apps analytics Final word # I hope this blog post has been a useful read to someone, as I said I wrote it as partial sample chapter for a publisher but they mucked me about so I though it was better off being on my blog rather than gathering virtual dust.\nThanks for reading\nUntil next time!\n","date":"December 9, 2017","externalUrl":null,"permalink":"/programming/2017-12-09-elmbook/","section":"Blog","summary":"Hi, its been a very long while since I last blogged, I promised this ages ago, its a partial sample chapter for a book I was planning on Elm but the publisher started to muck me about so it never happened.\n","title":"Building a single page application with Elm","type":"programming"},{"content":"","date":"December 9, 2017","externalUrl":null,"permalink":"/tags/elm/","section":"Tags","summary":"","title":"Elm","type":"tags"},{"content":"","date":"December 9, 2017","externalUrl":null,"permalink":"/tags/fable/","section":"Tags","summary":"","title":"Fable","type":"tags"},{"content":"So as promised here\u0026rsquo;s a post with more detail on the iOS designer provider that I presented as part of my talk at fsharpX 2017 The talk is entitled Expanding the Horizons of Mobile Development\nBackground # In iOS, user interfaces can be represented by storyboards, essentially a storyboard is a big ball of xml that is produced by a visual designer like Xcode or Xamarin\u0026rsquo;s Designer.\nThe basic premise of designer based user interfaces in .Net is the concept of a code behind file and and a partial class. For a C# Xamarin.iOS storyboard project something a bit like this will be present as a partial class:\nnamespace Designer { [Register (\u0026#34;DesignerViewController\u0026#34;)] partial class DesignerViewController { [Outlet] [GeneratedCodeAttribute (\u0026#34;iOS Designer\u0026#34;, \u0026#34;1.0\u0026#34;)] UIKit.UIButton submitButton { get; set; } void ReleaseDesignerOutlets () { if (submitButton != null) { submitButton.Dispose (); submitButton = null; } } } } With the user using another file with the same partial class to consume the controls generated by the designer:\nnamespace Designer { partial class DesignerViewController { public override void ViewDidLoad () { base.ViewDidLoad(); submitButton.TouchUpInside += (sender, e) =\u0026gt; { //Do stuff } } } I\u0026rsquo;ve always found the idea of partial class a bit of a cop out and never enjoyed having them around, which is why I do not miss them in F#. It does, however, pose a bit of problem with dealing with feature parity between C# and F# where designer files are generated as partial classes and you have to implement code in the other partial classes. In F# that whole aspect is entirely missing due to the lack of partial classes, which means anyone with a penchant for visual designers will be disappointed. (You can read more about how the iOS designer works in Xamarin.iOS) here\nIs there be a better way? # Of course! We can use the power of F# type providers! Although not for the feint hearted to create, F# type providers can be quite a boon to development time with their design time access to intellisense and tooltips based of some kind of underlying schema.\nDuring my time at Xamarin I did research work on lots of different areas of using F#, one of those was using F# type providers with designer tooling for iOS and Android. A few weeks back Miguel de Icaza kindly agreed to open source the work so that it could be enjoyed by all!\nIntroducing the iOS designer provider # Type providers come in two distinct forms, generative and erasing, generally most people use erasing providers where the code is erased at compile time to objects leaving only raw functionality. This is particularly useful for large schemas where information is generated on demand or the runtime representation is only data and methods. Generative providers produce real types that can be consumed and augmented in the normal fashion. Due to the nature of the iOS runtime and the way the Xamarin iOS works, generative type providers are needed.\nHow does it work? # The storyboard files within the current project are processed and any ViewControllers that have been assigned a name are processed by the type provider. Each control within the ViewController are also processed if they have been assigned a name. A property and disposal logic is generated as well as an abstract type for the controller.\nExample storyboard # Here is a sample storyboard from tvOS:\nThe ViewController is named ourViewController. The controls are equally imaginatively named as follows:\nlabel1 entry1 entry2 entry3 entry4 button1 Using the type provider # The first step is to instantiate the type provider, this effectively starts the process of finding the storyboards within the project and generating the types in the background:\ntype VCContainer = Xamarin.UIProvider The type bound above VCContainer can be called anything you wish I named it VCContainer because it contains all of the ViewControllers found in the storyboards.\nThe next step is to create and register a ViewController that was defined in one of the storyboards. Using auto-completion we can dot into the VCContainer to find any of the ViewControllers within the storyboard.\nHere the ViewController we saw in the screen shot above (ourViewController) is exposed with a base suffix ourViewControllerBase which is also an abstract type. We don\u0026rsquo;t want to allow the ViewController to be instantiated directly as the iOS runtime requires attributes on the exposed type and as we don\u0026rsquo;t yet have intrinsic type extensions on provided types we don\u0026rsquo;t have a great alternative apart from inheritance or augmentation.\n[\u0026lt;Register(VCContainer.ourViewControllerBase.CustomClass)\u0026gt;] type myViewController(ptr) = inherit VCContainer.ourViewControllerBase(ptr) Due to the way Xamarin.iOS works we have to place a Register attribute on the type so that the the iOS runtime can instantiate the correct type when the storyboard is constructed. Another feature of the iOS designer type provider is that the CustomClass name is exposed as a convenience so that you don\u0026rsquo;t have to remember or retype the name of the ViewController avoiding stringly typed hell. You can also see that ourViewControllerBase is called using the nativeint constructor: inherit VCContainer.ourViewControllerBase(ptr), this is how the storyboard infrastructure in iOS creates an instance of your ViewController`.\nWhen we come to wire up events and consume the controls we can do so very easily via auto completion. Say we want to wire up button1 so that when it is clicked we concatenate the contents of the Entry controls and place the text into label1 we could do so like this:\noverride x.ViewDidLoad () = base.ViewDidLoad () x.button1.PrimaryActionTriggered.Add( fun _ -\u0026gt; x.View.BackgroundColor \u0026lt;- UIColor.Blue x.label1.Text \u0026lt;- [ x.entry1.Text x.entry2.Text x.entry3.Text x.entyr4.Text ] |\u0026gt; String.concat \u0026#34; \u0026#34; ) You can see in ViewDidLoad we added an event handler to button1 and consume the relevant UI components.\nBenefits # The benefits of this approach depend of what angle you look from. From the C# approach at the start of this post there\u0026rsquo;s the reduced complexity of not having partial types and the designer generated parts being part of the project. You still get all the benefits of the UI controls being able to be consumed and available in auto completion etc. Compared to the current F# approach there\u0026rsquo;s less boiler plate because you dont have to manually add properties and Outlet attributes to your ViewController, with this approach there\u0026rsquo;s also the pitfall of stringly types which are extremly error prone. The final advantage is that if the storyboard is changed so that a control is deleted, the compilation will fail at design time rather than runtime.\nType providers provide a welcome safety net and boiler plate reduction for these type of scenarios.\nGenerated Code # For the curious the generated code in VCContainer looks like this:\npublic sealed class VCContainer { public abstract class ourViewControllerBase : UIViewController { public const string CustomClass = \u0026#34;ourViewController\u0026#34;; private UITextField __entry1; private UITextField __entry2; private UITextField __entry3; private UITextField __entyr4; private UIButton __button1; private UILabel __label1; [Outlet] public UITextField entry1 { get { return this.__entry1; } set { this.__entry1 = value; } } //entry2/3/4 etc } Here is the disposal logic generated for each control:\npublic void ReleaseDesignerOutlets () { if (this.__entry1 != null) { UnboxGeneric\u0026lt;IDisposable\u0026gt;(this.__entry1).Dispose (); } if (this.__entry2 != null) { UnboxGeneric\u0026lt;IDisposable\u0026gt;(this.__entry2).Dispose (); } if (this.__entry3 != null) { UnboxGeneric\u0026lt;IDisposable\u0026gt;(this.__entry3).Dispose (); } if (this.__entyr4 != null) { UnboxGeneric\u0026lt;IDisposable\u0026gt;(this.__entyr4).Dispose (); } if (this.__button1 != null) { UnboxGeneric\u0026lt;IDisposable\u0026gt;(this.__button1).Dispose (); } if (this.__label1 != null) { UnboxGeneric\u0026lt;IDisposable\u0026gt;(this.__label1).Dispose (); } if (this.__ourViewController != null) { UnboxGeneric\u0026lt;IDisposable\u0026gt;(this.__ourViewController).Dispose (); } } So whats next? # Since fsharpX Ive been super busy catching up with work so I\u0026rsquo;ve only managed to get the slides published for the talk and write this blog post, but fairly soon there will be a nuget package available that will allow the designer provider to be used for iOS, tvOS and watchOS. There is actually an experimental Android UI and fragment provider too, but that needs a little more work before its released on the general public :-)\nIf anyone is interested in the inner working of the type provider let me know and I\u0026rsquo;ll draft up some notes on that too. I know type providers are pretty mysterious and advanced concepts to a lot of people, generative providers doubly so.\nAs usual I love to get any feedback, comments and suggestions\u0026hellip;\nUntil next time!\n","date":"April 11, 2017","externalUrl":null,"permalink":"/programming/2017-04-11-i-want-to-tell-you-a-storyboard/","section":"Blog","summary":"So as promised here’s a post with more detail on the iOS designer provider that I presented as part of my talk at fsharpX 2017 The talk is entitled Expanding the Horizons of Mobile Development\n","title":"I want to tell you a storyboard","type":"programming"},{"content":"","date":"April 11, 2017","externalUrl":null,"permalink":"/tags/ios/","section":"Tags","summary":"","title":"Ios","type":"tags"},{"content":"","date":"April 11, 2017","externalUrl":null,"permalink":"/tags/xamarin/","section":"Tags","summary":"","title":"Xamarin","type":"tags"},{"content":"With the release of Elm 0.17 there were some fundamental changes to the Elm language. This post is my attempt to help those that may be struggling with these changes\nI\u0026rsquo;ve played with lots of new languages over the last year or so namely Elixir, Rust, and Elm. Elm and Elixir have been my favorites and I hope to cover those much more in this and future blog posts.\nSo whats new in Elm 0.17 why do I need to upgrade anything? # I\u0026rsquo;ll summarise here:\nSignals have been removed (Hence some upgrade is required if you had any code using signals.) Faster HTML renderer Libraries for geolocation, page visibility, and web sockets Generated JS is smaller and works with Google\u0026rsquo;s Closure Compiler Generated JS works with RequireJS and CommonJS Features in place for services like GraphQL and Elixir Phoenix Improved documentation at guide.elm-lang.org Helpful messages when decoding JSON fails The big things are that are going to throw a spanner in the works are Signals have been removed and the following packages have moved around:\nevancz/elm-html -\u0026gt; elm-lang/html evancz/elm-svg -\u0026gt; elm-lang/svg evancz/virtual-dom -\u0026gt; elm-lang/virtual-dom evancz/start-app -\u0026gt; elm-lang/html evancz/elm-effects -\u0026gt; elm-lang/core Be sure to read the official post on the subject for the full information.\nHow Do I upgrade? Are there any resources to help? # There are a few guides already that do help, the official upgrade plan is really useful as is migrating from elm 0.16 to 0.17 and I would advise that you read the official plan before this or any other guides. Hopefully my whistle stop tour of upgrading an existing package may be of help.\nExample Upgrade # Lets take the elm-sprite package as an example, its fairly simple with only a few dependencies.\nelm.package.json # Lets look at the elm.package.json file:\n{ \u0026#34;version\u0026#34;: \u0026#34;1.0.0\u0026#34;, \u0026#34;summary\u0026#34;: \u0026#34;Simple sprite rendering for elm-html\u0026#34;, \u0026#34;repository\u0026#34;: \u0026#34;https://github.com/Fresheyeball/elm-sprite.git\u0026#34;, \u0026#34;license\u0026#34;: \u0026#34;MIT\u0026#34;, \u0026#34;source-directories\u0026#34;: [ \u0026#34;src\u0026#34; ], \u0026#34;exposed-modules\u0026#34;: [ \u0026#34;Sprite\u0026#34; ], \u0026#34;dependencies\u0026#34;: { \u0026#34;elm-lang/core\u0026#34;: \u0026#34;3.0.0 \u0026lt;= v \u0026lt; 4.0.0\u0026#34; }, \u0026#34;elm-version\u0026#34;: \u0026#34;0.16.0 \u0026lt;= v \u0026lt; 0.17.0\u0026#34; } We need to update the dependencies for elm-lang/core to 4.0.1 \u0026lt;= v \u0026lt; 5.0.0 and elm-version to 0.17.0 \u0026lt;= v \u0026lt; 0.18.0. Pretty easy in terms of dependencies.\nSprite.elm # This ones pretty easy, the only thing to change is the module definition which is using an obsolete syntax:\nmodule Sprite (..) where Now becomes\nmodule Sprite exposing (..) Actually the Elm compiler does a fantastic job here by actually telling us what the problem is:\n-- SYNTAX PROBLEM --------------------------------------------------- Sprite.elm I ran into something unexpected when parsing your code! 1| module Sprite (..) where ^ I am looking for one of the following things: something like `exposing (..)` which replaced `where` in 0.17 whitespace So that was pretty painless, lets have a look at the example file: One.elm\nOne.elm # This ones a bit more tricky as there are signals involved and packages that have moved about.\nLets have a look at the changes needed.\nFirst of all lets address the obsolete where syntax:\nmodule One exposing (..) We need to remove the Signal, Html.Events and Effects imports packages, let\u0026rsquo;s remove these:\nimport Signal exposing (message, Address) import Html.Events exposing (on, targetValue) import Effects exposing (Effects, none) Now we need to address the changes in the Time package as fps is no longer available, we can use milliseconds instead.\nimport Time exposing (Time, millisecond) Now we need to adjust the StartpApp import and use Html.App instead.\nimport Html.App as Html So all in all the imports section will look like this:\nmodule One exposing (..) import Html exposing (..) import Html.App as Html import Time exposing (Time, millisecond) import Html.Attributes as A import Sprite exposing (..) import Array Action # The first thing we will tackle is the Action which flows though this application. It now looks like this:\ntype Action = Tick Time As Action has now been replaced with Msg so we need to change this to the following:\ntype Msg = Tick Time init # Next lets look at the init function its signature is slightly different now. Html.Program now starts the application\ninit : (model, Cmd msg) So rather than:\ninit : Sprite {} init = { sheet = \u0026#34;https://10firstgames.files.wordpress.com/2012/02/actionstashhd.png\u0026#34; , rows = 16 , columns = 16 , size = ( 2048, 2048 ) , frame = 0 , dope = idle } It will now become:\ninit : (Sprite {}, Cmd Msg) init = ( { sheet = \u0026#34;https://10firstgames.files.wordpress.com/2012/02/actionstashhd.png\u0026#34; , rows = 16 , columns = 16 , size = ( 2048, 2048 ) , frame = 0 , dope = idle } , Cmd.none) view # This is what view looks like:\nview : Address Action -\u0026gt; Sprite {} -\u0026gt; Html view address s = let onInput address contentToValue = on \u0026#34;input\u0026#34; targetValue (message address \u0026lt;\u0026lt; contentToValue) in div [] [ node \u0026#34;sprite\u0026#34; [ A.style (sprite s) ] [] ] At first glance this looks a bit more complex but when you look at the code a little more you come to realise that the onInput function is not even used anymore this is just dead code. So no all that remains is to change view to match the new Elm 0.17 architecture, so instead of Address Action -\u0026gt; Sprite {} -\u0026gt; Html it will now be Sprite {} -\u0026gt; Html Msg as Address and Action are now no longer needed.\nview : Sprite {} -\u0026gt; Html Msg view s = div [] [ node \u0026#34;sprite\u0026#34; [ A.style (sprite s)] [] ] update # Ok, now for update lets have a look at that:\nupdate : Action -\u0026gt; Sprite {} -\u0026gt; ( Sprite {}, Effects Action ) update action s = let s\u0026#39; = case action of Tick _ -\u0026gt; advance s in ( s\u0026#39;, none ) update now has a signature of msg -\u0026gt; model -\u0026gt; (model, Cmd msg) so all we really have to do is replace Action with Msg, and Effects Action with Cmd Msg. Finally I change the return to: Cmd.none which was previously Effects.none.\nupdate : Msg -\u0026gt; Sprite {} -\u0026gt; (Sprite {}, Cmd Msg) update action s = let s\u0026#39; = case action of Tick _ -\u0026gt; advance s in ( s\u0026#39;, Cmd.none ) subs # This part is new, with the old 0.16 based version there was a signal which was mapping time to a sprite update:[ Signal.map Tick (fps 30) ]. Now that we will be using subscriptions this is a simple function like this:\nsubs : Sprite {} -\u0026gt; Sub Msg subs model = Time.every (millisecond * 33) Tick So every 33 milliseconds (30 frames per second) we are sending a Tick command to the update function.\napplication start # The last part is the application start, heres what it looks like in 0.16:\napp : StartApp.App (Sprite {}) app = StartApp.start { view = view , update = update , init = ( init, none ) , inputs = [ Signal.map Tick (fps 30) ] } main : Signal Html main = app.html You can see the signal I talked about above. This whole section has now become a lot simpler in Elm 0.17.\nmain : Program Never main = Html.program { view = view , update = update , init = init , subscriptions = subs } StartApp has now become Html.App which we aliased to Html at the beginning import Html.App as Html and we use the program function to feed in all the functions we just declared.\nOk, we\u0026rsquo;re all done, hopefully someone found this useful!\nOne tip I can give is to update the function signatures for each function first and it should make things a little clearer on what you need to do.\nUntil next time!\n","date":"June 30, 2016","externalUrl":null,"permalink":"/programming/2016-06-30-elmtastic-updates/","section":"Blog","summary":"With the release of Elm 0.17 there were some fundamental changes to the Elm language. This post is my attempt to help those that may be struggling with these changes\n","title":"Elmtastic Updates","type":"programming"},{"content":"","date":"June 30, 2016","externalUrl":null,"permalink":"/tags/functional-programming/","section":"Tags","summary":"","title":"Functional-Programming","type":"tags"},{"content":"So as promised here\u0026rsquo;s a guide to creating your first fable |\u0026gt; fuse application\nI\u0026rsquo;ve made a template available so this can be tried out quickly and easily, I\u0026rsquo;ll run through the requirements and describe whats in the template.\nRequirements # First of all here are the requirements.\nYeoman # As fable uses npm modules for dependencies fable |\u0026gt; fuse template will also be based on them. I have used Yeoman as it\u0026rsquo;s very flexible and works nicely.\nSo what you need to get Yeoman is:\nnpm install -g yo And then to install the fable |\u0026gt; fuse template for Yeoman:\nnpm install -g generator-fable-fuse Nice and easy.\nFable # Installing Fable is covered in the Fable documentation, but it\u0026rsquo;s essentially just another npm install:\nnpm install -g fable-compiler Fuse # Fuse can be downloaded at their site here\nThat\u0026rsquo;s pretty much all the requirements, let\u0026rsquo;s try this out!\nCreating a fable |\u0026gt; fuse application? # OK, so with everything installed how do you get going?\nThis is really easy:\nmkdir fable-fuse-test cd fable-fuse-test yo fable-fuse And then follow the prompts, here\u0026rsquo;s a asciinema session showing the process:\nNow that the template has been created let\u0026rsquo;s have a look at whats inside.\nProject Structure # The structure of a fable |\u0026gt; fuse application is as follows (Using a project name of test):\n├── App │ ├── MainView.ux │ └── test.unoproj ├── build.bat ├── build.sh ├── node_modules │ ├── fable-core │ ├── fable-fuse │ └── fable-import-fetch ├── package.json └── src ├── fableconfig.json ├── test.fs └── test.fsproj Lets go through the root files first\npackage.json # This file has the dependencies for fable |\u0026gt; fuse. Currently these are:\nfable-core : This has the main definitions for Fable. fable-fuse : This has the bindings for the Fuse JavaScript API\u0026rsquo;s. fable-import-fetch : This has the F# bindings for JavaScript Fetch API. build.sh / build.bat # These two files contain the script to transpile the F# source into JavaScript. So upon typing ./build.sh the F# files will be transpiled into JavaScript and placed into the App/js folder. In addition Fable will continue to watch the F# files and transpile the files if they change so you get realtime updating of the fuse application.\nNow the directories:\nApp # The App directory has all the necessary files that Fuse requires to build\ntest.unoproj # This is the project file for Fuse, it has settings for the different platforms and controls which gets embedded in the application.\nMainView.ux # This is the main view markup file for the user interface.\nsrc # The src folder has all the F# source files that Fable transpiles into JavaScript.\nfableconfig.json # This is the configuration file for Fable which allows you to run scrip before or after compilation and set various defaults.\ntest.fs # This is the main source file for F# and is the same as the one you saw in the previous post:\nnamespace App open Fable.Core open Fuse open Fable.Import open Fable.Import.Fetch module test = let data = Observable.create() promise { let! req = GlobalFetch.fetch (Url \u0026#34;http://az664292.vo.msecnd.net/files/ZjPdBhWNdPRMI4qK-colors.json\u0026#34;) let! json = req.json () do (data.value \u0026lt;- json) } |\u0026gt; ignore This is fairly easy to follow, the promise block is a custom computation expression that allows each successful JavaScript promise to execute before the next promise is ran. In this example GlobalFetch.fetch and req.json() both return JavaScript promises. The promise block runs the GlobalFetch.fetch function and if it succeeds it runs the req.json() function. If that too is successful then the observable value data is updated to the resulting json data.\nnode-modules # These are our dependencies, there\u0026rsquo;s fable-core which is required by Fable, also included are fable-fuse which are the F# bindings to the Fuse JavaScript libraries, and fable-import-fetch which is the F# bindings for the Fetch JavaScript API.\nRunning # To get a fable |\u0026gt; fuse application running all you have to do is run the build script ./build which transpiles the F# files into JavaScript and then watches for any updates to the F# files. Running a Fuse application is also really easy, you can do this from within Atom via the plugin, or sublime via that plugin, or simply just run fuse preview ./App/ from the project root.\nAny Problems? # If you have any problems with the Yeoman generator for fable |\u0026gt; fuse then please log an issue on its GitHub repo: generator-fable-fuse. If you have any issues with the fable-fuse module itself then please got an issue on its GitHub repo: fable-fuse.\nIf you have an improvements or suggestions then a PR is very welcome too!\nWhats next? # If there is enough interest around using fable |\u0026gt; fuse I\u0026rsquo;ll port some of the more intricate samples from the Fuse examples over to fable |\u0026gt; fuse and also create a GitHub site with all the content relating to it.\nLet me know what you think!!\nUntil next time!\n","date":"June 7, 2016","externalUrl":null,"permalink":"/programming/2016-07-06-fable-fuse-template/","section":"Blog","summary":"So as promised here’s a guide to creating your first fable |\u003e fuse application\n","title":"Creating fuse applications with fable","type":"programming"},{"content":"","date":"June 7, 2016","externalUrl":null,"permalink":"/tags/fuse/","section":"Tags","summary":"","title":"Fuse","type":"tags"},{"content":"","date":"June 7, 2016","externalUrl":null,"permalink":"/tags/javascript/","section":"Tags","summary":"","title":"Javascript","type":"tags"},{"content":"In this post Im going to be introducing something new and exciting the combination of F#, Fable and Fuse.\nThis is the start of an exciting new series on using F# as a transpiling language to \u0026ldquo;light the Fuse\u0026rdquo; (pun intended) on new platform and opportunities for F#.\nOK, so now for the introductions\u0026hellip;\nWhat is Fable # Put simply Fable is a transpiler for F# that turn F# into JavaScript. Lets face it JavaScript is really pervasive but not everyone wants to write it. You can read about how Fable works here\nWhat is fuse? # Taken from their site this is what fuse is in a nutshell:\nFUSE IS FOR MOBILE APP DESIGNERS AND DEVELOPERS # Create and update the look and feel for native apps in real time on multiple devices simultaneously.\nFuse is a set of tools (currently in beta) that makes design and developing native mobile apps for iOS and Android fast, easy and fun. Fuse is free, and we’re actively working towards making it Open Source as well.\nFuse introduces UX Markup - a XML-based language for creating truly native, data-driven, responsive, smoothly animated and highly interactive experiences, while sharing most of the code between iOS and Android. UX is easy to learn and incredibly powerful.\nFuse is fast. Based on Uno, a language which compiles down to pure C++ code, and seamlessly interops with Objective-C (iOS) and Java (Android) where needed. The UI is rendered using native platform controls, OpenGL or a combination (best of both worlds).\nFor business logic, Fuse runs JavaScript on a separate thread on both iOS and Android, so your UI is fast and responsive no matter what it is doing. Fuse lets you call seamlessly into C++, Java and Objective-C libraries through Uno when you need it.\nYou can read about Fuse in more depth on their site here, you really should take a look.\nIntroducing Fable |\u0026gt; Fuse # Fable-Fuse is a set of packages that allows you use power of F# with Fuse.\nFirst of all lets have a little look at some declarative UI with Fuse.\nThis is taken from the original Fuse Sample titled Parsing JSON fetched over HTTP which is located here\n\u0026lt;App Theme=\u0026#34;Basic\u0026#34; Background=\u0026#34;#eee\u0026#34;\u0026gt; \u0026lt;DockPanel\u0026gt; \u0026lt;StatusBarBackground Dock=\u0026#34;Top\u0026#34; /\u0026gt; \u0026lt;BottomBarBackground Dock=\u0026#34;Bottom\u0026#34; /\u0026gt; \u0026lt;ScrollView\u0026gt; \u0026lt;Grid ColumnCount=\u0026#34;2\u0026#34;\u0026gt; \u0026lt;JavaScript\u0026gt; var Observable = require(\u0026#34;FuseJS/Observable\u0026#34;); var data = Observable(); fetch(\u0026#39;http://az664292.vo.msecnd.net/files/ZjPdBhWNdPRMI4qK-colors.json\u0026#39;) .then(function(response) { return response.json(); }) .then(function(responseObject) { data.value = responseObject; }); module.exports = { data: data }; \u0026lt;/JavaScript\u0026gt; \u0026lt;Each Items=\u0026#34;{data.colorsArray}\u0026#34;\u0026gt; \u0026lt;DockPanel Height=\u0026#34;120\u0026#34; Margin=\u0026#34;10,0\u0026#34;\u0026gt; \u0026lt;Panel DockPanel.Dock=\u0026#34;Top\u0026#34; Margin=\u0026#34;10\u0026#34; Height=\u0026#34;30\u0026#34;\u0026gt; \u0026lt;Rectangle Layer=\u0026#34;Background\u0026#34; CornerRadius=\u0026#34;10\u0026#34; Fill=\u0026#34;#fff\u0026#34;/\u0026gt; \u0026lt;Text Value=\u0026#34;{colorName}\u0026#34; TextAlignment=\u0026#34;Center\u0026#34; Alignment=\u0026#34;Center\u0026#34; /\u0026gt; \u0026lt;/Panel\u0026gt; \u0026lt;Rectangle Layer=\u0026#34;Background\u0026#34; CornerRadius=\u0026#34;10\u0026#34; Fill=\u0026#34;{hexValue}\u0026#34;/\u0026gt; \u0026lt;/DockPanel\u0026gt; \u0026lt;/Each\u0026gt; \u0026lt;/Grid\u0026gt; \u0026lt;/ScrollView\u0026gt; \u0026lt;/DockPanel\u0026gt; \u0026lt;/App\u0026gt; You can see the UI mark-up is concise and easy to follow, also notice the JavaScript element which can be in-line, as shown here, or placed in a separate file. We will place this in a separate file so that we can transpile from F# using Fable.\nnamespace Program open Fable.Core open Fuse open Fable.Import open Fable.Import.Fetch module HttpJson = let data = Observable.create() promise { let! req = GlobalFetch.fetch (Url \u0026#34;http://az664292.vo.msecnd.net/files/ZjPdBhWNdPRMI4qK-colors.json\u0026#34;) let! json = req.json () do (data.value \u0026lt;- json) } |\u0026gt; ignore You can see here there is a custom computation expression that allows you to use JavaScrip promises, you could also use a pipeline oriented definition too, like this:\nGlobalFetch.fetch (Url \u0026#34;http://az664292.vo.msecnd.net/files/ZjPdBhWNdPRMI4qK-colors.json\u0026#34;) |\u0026gt; Promise.success (fun resp -\u0026gt; resp.json()) |\u0026gt; Promise.success (fun json -\u0026gt; data.value \u0026lt;- json) |\u0026gt; ignore Or if you really really wanted to you could integrate this into an F# async with a little helper type extension:\nmodule AsyncExtensions = type Microsoft.FSharp.Control.AsyncBuilder with member x.Bind(p, f) = async.Bind (Async.AwaitPromise(p), f) This would allow you to use a promise with an ordinary let!\nasync { let! req = GlobalFetch.fetch \u0026#34;http://az664292.vo.msecnd.net/files/ZjPdBhWNdPRMI4qK-colors.json\u0026#34; let! json = req.json () do (data.value \u0026lt;- json) } |\u0026gt; Async.Start Anyway I digress, needless to say there are various options with promises and how to handle them with Fable |\u0026gt; Fuse.\nWhat\u0026rsquo;s more because Fuse and Fable are real-time you can edit the UX definitions and it the application in real-time across multiple devices!\nSo how do I get started with Fable |\u0026gt; Fuse ? # Well you\u0026rsquo;ll have to hold your horses, I was so excited to share this introductory post I haven\u0026rsquo;t wrote that part yet. The package I\u0026rsquo;m working on is still private while I finalise things and make it really easy and friendly to create applications with Fable |\u0026gt; Fuse.\nStay tuned as there will be more in this series next week as I discuss the more technical aspects and how to create your first Fable |\u0026gt; Fuse application.\nIf there is enough interest I will also live stream this on my livecoding.tv channel, please subscribe.\nA really big thanks to Alfonso Garcia-Caro (@alfonsogcnunez) creator of Fable for answering all my annoying questions. And Lars Thomas Denstad (@cocporn) for help creating the Fuse API bindings for Fable.\nUntil next time!\n","date":"June 3, 2016","externalUrl":null,"permalink":"/programming/2016-06-03-light-the-fuse/","section":"Blog","summary":"In this post Im going to be introducing something new and exciting the combination of F#, Fable and Fuse.\n","title":"Light The Fuse","type":"programming"},{"content":"","date":"May 31, 2016","externalUrl":null,"permalink":"/tags/microsoft/","section":"Tags","summary":"","title":"Microsoft","type":"tags"},{"content":"So today was my last day at Xamarin.\nI have decided to move on and have some fabulous new adventures. I don\u0026rsquo;t know quite where this will lead me yet but its going to be exciting finding out!\nWhy? you may ask. # I\u0026rsquo;ve been working for Xamarin since 22nd October 2013 on F# tooling and engineering. Over a period of time it\u0026rsquo;s very easy to become comfortable and complacent and no longer be challenged. I have a hacker mentality, I need to create new things, crack difficult problems, experiment. There are various other reasons too, but with the Microsoft acquisition now seemed as good a time as any to say goodbye and move on.\nWhat\u0026rsquo;s Next? # So in a nutshell, I will also be writing, live-streaming and working on my own new and wonderful things to challenge myself and push my boundaries.\nI\u0026rsquo;ll once again available for some consulting work too, so feel free to ping me.\nUntil next time!\n","date":"May 31, 2016","externalUrl":null,"permalink":"/programming/2016-05-31-new-adventures/","section":"Blog","summary":"So today was my last day at Xamarin.\n","title":"New Adventures","type":"programming"},{"content":"A few weeks back I posted on Twitter that I was experimenting with flame graphs, In this post I will share how this was accomplished.\nRequirements # First of all the requirements, I\u0026rsquo;m assuming your using a Mac just as I am. If you are not then you might be able to use x-perf for windows check out summarizing-xperf-cpu-usage-with-flame-graphs for information in that area. I don\u0026rsquo;t really use Windows that often but if I do happen to try this out on Windows then I\u0026rsquo;ll pop back here and update this post.\nOK, so back to requirements, a Mac, Mono Installation, Xcode installed so that you can use Instruments to collect trace information, clone the FlameGraph repo itself:\ngit clone https://github.com/brendangregg/FlameGraph The FlameGraph repo is a bunch of scripts to help process the trace data and produce an svg. You can read more about FlameGraphs here: http://www.brendangregg.com/flamegraphs.html\nAOT the framework # Next step is to AOT compile all the mono runtime assemblies, you can do this by running the following commands from your mono installation. For me this would be:\ncd /Library/Frameworks/Mono.framework/Versions/4.4.0/lib/mono/ for i in `find gac -name \u0026#39;*dll\u0026#39;` */mscorlib.dll; do mono --aot $i done AOT Your app # Now you need to AOT you applications files with the same command\nmono --aot myApp.exe Attaching Instruments # Now you are ready to run your app and attach Instruments or have Instruments launch your app.\nFor simplicity I opted to add this to the beginning of my app:\nprintfn \u0026#34;Press any key to start\u0026#34; Console.ReadKey() |\u0026gt; ignore That way I could just launch my app (noting the process id) and use the process browser within Instruments to attach.\nNow launch Instruments and select the Time Profiler template:\nYou can tweak the sampling interval with the settings on the right, in my example below I was using 40us because it was a really fast executing demo.\nUse the process browser in Instruments to choose the mono process running my app e.g. mono (16314).\nNow that instruments is attached hit the big red record button and hit any key on you app to start collecting data. When you are finished just hit the top button in Instruments.\nYou should end up with something similar to this, I used a few cycles of my 68000 emulator to get this data:\nExporting The Data # Exporting the data is pretty easy, use expand all on a node in the collected data using Cmd cLick, you can also add filters to the data, I used Atari in the screen-shot above to constrain the output to nodes that contained Atari. Now select export from the instrument menu: Producing the FlameGraph # Now you can open a move to the flamegraph repo that you cloned earlier and execute the following command replacing myoutput.csv|svg with your input/outputs.\n./stackcollapse-instruments.pl myoutput.csv | ./flamegraph.pl \u0026gt;myoutput.svg You should now have a funky FlameGraph!\nYou could quite easily post process the csv output to clean up the mangled names that are a result of the AOT process.\nUntil next time \u0026hellip;\n","date":"May 29, 2016","externalUrl":null,"permalink":"/programming/2016-05-29-flame-on/","section":"Blog","summary":"A few weeks back I posted on Twitter that I was experimenting with flame graphs, In this post I will share how this was accomplished.\n","title":"flame on","type":"programming"},{"content":"","date":"May 29, 2016","externalUrl":null,"permalink":"/tags/performance/","section":"Tags","summary":"","title":"Performance","type":"tags"},{"content":"I wanted to briefly talk about what I\u0026rsquo;ve been up to lately. I\u0026rsquo;ve been doing a spot of compiler hacking, working on improving Type Providers and generally tinkering with concepts relating to macros etc. I\u0026rsquo;ve also been tinkering with the Elixir and Rust which I quite like too.\nSo, emulators, that\u0026rsquo;s what I was going to talk about today. Lately I\u0026rsquo;ve been thinking about emulators, reading about them, and also watching streams about writing them. I\u0026rsquo;ve always been interested in emulators but never sat down and looked at the theory or written one. So I\u0026rsquo;ve started live streaming creating one on LiveCoding.tv which is apparently where you go to:\nWatch coders code products live and hang out with them\nSo I\u0026rsquo;ve started live coding an Atari ST emulator with its 68000 CPU using LiveCoding.tv. I\u0026rsquo;ve chosen an Atari St because it was one of the computers that I grew up with and the computer I first started to program on. I have many many fond memories of it. I may also stream about building other types of emulators not just the 68000. There are plenty of interesting hardware systems and processor types that I grew up with. I was thinking of building an arcade emulator such as Capcom Play System but the technical data is a bit more difficult to get hold of.\nIn some respects emulator coding is quite tricky, a lot of research and digging is required to get the relevant information you need, and a lot of grunt work inputting data can also be involved too. In short its not a trivial task, at least it isn\u0026rsquo;t if you want to do one properly. I cant promise it will be super interesting, or even done properly, but it will be raw coding warts and all :-)\nYou can find my channel here Building Emulators with F# I normally stream a couple of days a week, currently Wednesday and Friday 17:00 GMT but I also may also stream at other times too.\nUntil next time \u0026hellip;\n","date":"March 6, 2016","externalUrl":null,"permalink":"/programming/2016-03-06-building-emulators-in-fsharp/","section":"Blog","summary":"I wanted to briefly talk about what I’ve been up to lately. I’ve been doing a spot of compiler hacking, working on improving Type Providers and generally tinkering with concepts relating to macros etc. I’ve also been tinkering with the Elixir and Rust which I quite like too.\n","title":"Building Emulators in FSharp","type":"programming"},{"content":"Ive been meaning to write this post for ages but Ive only just found the time.\nOn the 3rd of September my good friend Ryan Riley (aka @panesofglass) emailed me letting me know I had won a Community for F# hero award for 2015!\nI want to say a big thank you to everyone who voted for me. Its really nice to be recognized for the things that you do.\nAnyway, this is just a quick post to say thanks, and share a cup of coffee :-)\nUntil next time \u0026hellip;\n","date":"October 25, 2015","externalUrl":null,"permalink":"/programming/2015-10-25-f-community-hero/","section":"Blog","summary":"Ive been meaning to write this post for ages but Ive only just found the time.\nOn the 3rd of September my good friend Ryan Riley (aka @panesofglass) emailed me letting me know I had won a Community for F# hero award for 2015!\n","title":"F# Community Hero","type":"programming"},{"content":"I\u0026rsquo;ve finally been awarded a Microsoft MVP its been quite a long time coming. Ive been nominated for MVP every year since July 2011 but until now I have been unsuccessful. On the 1st of July I received an email congratulating me that I had been successful on my MVP nomination.\nI am now officially a Microsoft Most Valued Professional! # I was first nominated in July 2011 which seems like an age ago. I have been nominated various times each year since then. There are various activities that I have been involved in over the that last four or so years in F#. Here are the ones that I feel particularly proud of:\nBlogging # I write about various F# topics that I find interesting. They are usually technical in nature or just what I find interesting or challenging at the time. If I can help someone else figure something out from one of my posts, or bootstrap an interesting new project, then all the better!\nRenovating F# addin for MonoDevelop # When I first started using F# on OSX the MonoDevelop F# addin was pretty derelict, it hadn\u0026rsquo;t been updated for sometime and didn\u0026rsquo;t even compile. It took quite a learning curve but I soon got up to speed and managed to improve things bit by bit.\nEditor support and general tooling can take a lot of time and effort, you can often get stuck on things with no clue on how to resolve something without digging deep within the F# compiler. Im sure Don Syme has a spam filter on my emails by now :-)\nF# on IOS # Around January 2013 I managed to get F# working on iOS devices, you can read some of the details in part one and part two. It was really exciting to get a shinny new platform availably for F#!\nEdge.fs # Before the FSharp.Compiler.Service even came to light I heard about Edge.js and wanted to have F# support, so in May 2013 I set about implementing it. I ended up playing around in the compiler trying to figure how things worked. You can read all about that here.\nF# for ScriptCS # Around June 2013 ScriptCS was getting a great deal of publicity on Twitter so I decided it was about time F# should be part of it. This was also another opportunity to start hacking in the compiler again, ultimately this ended up with better programmatic REPL support added to the F# compiler.\nCreation of F# Compiler Service # As part of working on the F# plugin for MonoDevelop it came to light that better support was needed for the F# compiler so that better tool integration could be built. I worked to try and consolidate different areas of the the compiler that were being used by disparate tooling at the time. This eventually lead to working with Don Syme to create FSharp.Compiler.Service. Over time this repository has been improved and lead to even better tooling for all editors and IDE\u0026rsquo;s that use it.\nFSharp.Core nuget package # I petitioned for some time to get better PCL support for F#, namely the quite common profiles 78 (.NET Framework 4.5, Windows 8, Windows Phone Silverlight 8) and 259 (.NET Framework 4.5, Windows 8, Windows Phone 8.1, Windows Phone Silverlight 8). Once the extra profiles were brought to fruition I created a nuget package to deploy them for easy consumption.\nF# Community Bad Ass # This is one of my favorites: In May 2014 I was awarded the F# community Badass award at Functional Londoners:\n@7sharp9 @fsharporg \u0026quot;For services to the F# community in open source, cross-platform IDE, mobile and runtime tools\u0026quot;. Badass #1\n\u0026mdash; Don Syme (@dsymetweets) May 9, 2014 I wish I had got an official trophy for that, it would have looked really good on the mantlepiece! :-)\nUntil next time \u0026hellip;\n","date":"July 16, 2015","externalUrl":null,"permalink":"/programming/2015-07-16-reckoning-day/","section":"Blog","summary":"I’ve finally been awarded a Microsoft MVP its been quite a long time coming. Ive been nominated for MVP every year since July 2011 but until now I have been unsuccessful. On the 1st of July I received an email congratulating me that I had been successful on my MVP nomination.\n","title":"Reckoning day","type":"programming"},{"content":"","date":"July 12, 2015","externalUrl":null,"permalink":"/tags/elixir/","section":"Tags","summary":"","title":"Elixir","type":"tags"},{"content":"Sit down, strap in, and prepare for take off, we\u0026rsquo;re going Meta-Matic!\nWe\u0026rsquo;re going to be exploring some metaprogramming magic in F#. Transforming its abstract syntax tree (AST) into another languages AST before executing it in another virtual machine, Exciting!!\nBackground # Nothing really known can continue to be acutely fascinating\nFirst a little bit of background. There are three main forms of metaprogramming in F#, although you could call Type Providers a form of metaprogramming albeit limited in scope. I shall now briefly describe each below.\nQuotations # According to MSDN:\nCode quotations, a language feature that enables you to generate and work with F# code expressions programmatically. This feature lets you generate an abstract syntax tree that represent\u0026rsquo;s F# code. The abstract syntax tree can then be traversed and processed according to the needs of your application\nQuotations are fairly useful for translating some parts of the F# language to another language. They can depend on Reflection and having compiled DLLs for any types referenced in the quoted expression. They also have limitations like types are not supported, neither are variables that escape their quoted scope, and also generic definitions are not supported. They cant really be used for staged metaprogramming where you may need to refer to the full language and its constructs.\nCode Example # \u0026lt;@ [for i in 1 .. 10 do yield i * i] @\u0026gt; A textual representation of what this looks like is as follows:\nQuotations.Expr\u0026lt;int list\u0026gt; = Call (None, ToList, [Call (None, CreateSequence, [Call (None, Delay, [Lambda (unitVar, Call (None, Map, [Lambda (i, Call (None, op_Multiply, [i, i])), Call (None, op_Range, [Value (1), Value (10)])]))])])]) Untyped AST # There\u0026rsquo;s also the untyped AST that can be manipulated in some ways although it\u0026rsquo;s not exactly pleasant to do.\nCode Example # [for i in 1 .. 10 do yield i * i] A textual representation of what this looks like is as follows:\n[SynModuleOrNamespace ([Test {idRange = Test.fs (1,0--1,0); idText = \u0026#34;Test\u0026#34;;}],true, [DoExpr (SequencePointAtBinding Test.fs (1,0--1,33) {...}, ArrayOrListOfSeqExpr (false, CompExpr (true,{contents = true;}, ForEach (SequencePointAtForLoop Test.fs (1,1--1,20) {...}, SeqExprOnly false,true, Named (Wild Test.fs (1,5--1,6) {...}, i {idRange = Test.fs (1,5--1,6); idText = \u0026#34;i\u0026#34;;},false,null, Test.fs (1,5--1,6) {...}), App (NonAtomic,false, App (NonAtomic,true, Ident op_Range {idRange = Test.fs (1,12--1,14); idText = \u0026#34;op_Range\u0026#34;;}, Const (Int32 1, Test.fs (1,10--1,11) {...}), Test.fs (1,10--1,14) {...}), Const (Int32 10, Test.fs (1,15--1,17) {...}), Test.fs (1,10--1,17) {...}), YieldOrReturn ((true, false), App (NonAtomic,false, App (NonAtomic,true, Ident op_Multiply {idRange = Test.fs (1,29--1,30); idText = \u0026#34;op_Multiply\u0026#34;;}, Ident i {idRange = Test.fs (1,27--1,28); idText = \u0026#34;i\u0026#34;;}, Test.fs (1,27--1,30) {...}), Ident i {idRange = Test.fs (1,31--1,32); idText = \u0026#34;i\u0026#34;;}, \u0026lt;snip\u0026gt; As you can see it\u0026rsquo;s a bit of a handful, I even omitted some of the range statements too. It\u0026rsquo;s possible to manipulate this but it\u0026rsquo;s not exactly easy without a lot of helper functions and patience. Maybe I can cover this in a future post as it has its own use case in terms of F# manipulation and recompilation, but that\u0026rsquo;s a whole other story. Outside of the compiler it\u0026rsquo;s mainly used for code analysis type operations.\nTyped AST / Typed Expressions # There\u0026rsquo;s also the Typed AST (TAST) or Typed Expressions that fully represent\u0026rsquo;s the F# language. Like quotations, they give you a detailed view of checked/resolved/typed F# expressions. They have no dependency on reflection and do not require any on disk assemblies. They can be generated for solutions that that contain errors, and include all F# language constructs including those used in FSharp.Core.\nCode Example # [for i in 1 .. 10 do yield i * i] A textual representation of what this looks like is as follows:\n[Entity (Test, [InitAction Call (null,val toList,[],[int], [Call (null,val seq,[],[type int], [Call (null,val delay,[],[int], [Lambda (val unitVar, Call (null,val map,[], [type int; type int], [Lambda (val i, Call (null,val op_Multiply,[], [type int; type int; type int],[Value val i; Value val i])); Call (null,val op_Range,[],[type int], [Const (1,type int); Const (10,type int)])]))])])])])] In this example and the preceding one you will notice that they both begin with either an Entity in this example, and SynModuleOrNamespace in the previous. This is because dealing with the full F# metadata means we have access to the full range of F# syntactic constructs. Entities also sit within another file based construct that I have omitted from these examples for the benefit of clarity.\nLet\u0026rsquo;s get to work # For the purpose of this blog post we are going to use a basic F# module with a single function. We will be performing all sorts of black magic so it\u0026rsquo;s best to constrain this from the outset. (Note: Im joking here)\nThere is nothing more beautiful than an elegant mathematical proof.\nThe module and function we will be using is very simple and looks like this:\nmodule Test let square x = x * x Retrieving the TAST # First things first how do we get hold of the TAST?\nWe will need the FSharp.Compiler.Service package that can be installed from your IDE or favorite command line tool such as nuget:\nnuget install FSharp.Compiler.Service Now that it\u0026rsquo;s installed we can start using it.\n#r \u0026#34;../packages/FSharp.Compiler.Service.0.0.89/lib/net45/FSharp.Compiler.Service.dll\u0026#34; open System open System.IO open Microsoft.FSharp.Compiler.SourceCodeServices let testModule = \u0026#34;\u0026#34;\u0026#34;module Test let square x = x * x \u0026#34;\u0026#34;\u0026#34; let file = __SOURCE_DIRECTORY__ + \u0026#34;/Test.fs\u0026#34; File.WriteAllText(file, testModule) let checker = FSharpChecker.Create(keepAssemblyContents=true) let options = checker.GetProjectOptionsFromCommandLineArgs(\u0026#34;Test\u0026#34;, [|\u0026#34;-o:Test.dll\u0026#34;;\u0026#34;-a\u0026#34;;file|]) let checkProjectResults = checker.ParseAndCheckProject(options) |\u0026gt; Async.RunSynchronously In order to examine the TAST we have to make sure that when creating the FSharpChecker we set the optional parameter keepAssemblyContents to true, this lets you examine the typed AST in full.\nExamining the TAST # Examining checkProjectResults will yield several properties: AssemblyContents, AssemblySignature, Errors, HasCriticalErrors, and ProjectContext. Most of those are pretty obvious, if we examine Errors and HasCriticalErrors we could decide to abandon further processing but for the purposes of this example we are going to assume that there are no errors.\nThe property that we are interested in is AssemblyContents this essentially contains a list of ImplementationFiles, and an ImplementationFile has a list of FSharpImplementationFileDeclaration.\nLet\u0026rsquo;s go ahead and examine what we currently have:\ncheckProjectResults.AssemblyContents.ImplementationFiles.Head.Declarations This will yield a similar structure to what you saw above in the list comprehension.\nI have shortened the the Microsoft.FSharp.Core.* prefixes to save space.\n[Entity (Test, [MemberOrFunctionOrValue (val square,[[val x]], Call (null,val op_Multiply,[], [type int; type int; type int],[Value val x; Value val x]))])] The FSharpImplementationFileDeclaration is a discriminated union type that is defined as follows:\ntype FSharpImplementationFileDeclaration = | Entity of FSharpEntity * FSharpImplementationFileDeclaration list | MemberOrFunctionOrValue of FSharpMemberOrFunctionOrValue * FSharpMemberOrFunctionOrValue list list * FSharpExpr | InitAction of FSharpExpr With this in mind we could now traverse the tree with something like this:\nlet rec processDecl decl = match decl with | Entity(ent, declList) -\u0026gt; printfn \u0026#34;Entity\u0026#34; declList |\u0026gt; List.iter processDecl | InitAction(expr) -\u0026gt; printfn \u0026#34;Init Action\u0026#34; | MemberOrFunctionOrValue(memb, curriedParameterGroups, expr)-\u0026gt; printfn \u0026#34;Member\u0026#34; printfn \u0026#34;Expressions: %A\u0026#34; expr for implFile in checkProjectResults.AssemblyContents.ImplementationFiles do for decl in implFile.Declarations do processDecl decl This would yield similar output to what we saw before although we are now getting a feel for the structure of the tree.\nEntity:Test Member:square Expressions: Call (null,val op_Multiply,[], [type int; type int; type int],[Value val x; Value val x]) Preparing To transform # Ok, so we know a little about the TAST structure and how to navigate it, now we can look to transform it. So what can we transform the F# TAST to?\nAbsolutely anything we want, but for the purposes of this article let\u0026rsquo;s choose something interesting \u0026hellip; like Elixir.\nElixir language # So what is Elixir? I\u0026rsquo;ll quote the description from the Elixir website as this post is not about describing the Elixir language.\nElixir is a dynamic, functional language designed for building scalable and maintainable applications.\nElixir leverages the Erlang VM, known for running low-latency, distributed and fault-tolerant systems, while also being successfully used in web development and the embedded software domain.\nElixir AST structure # The Elixir AST structure is described like this:\n{atom | tuple, list, list | atom} The first element is an atom or another tuple in the same representation; The second element is a keyword list containing metadata, like numbers and contexts The third element is either a list of arguments for the function call or an atom. When this element is an atom, it means the tuple represent\u0026rsquo;s a variable. Elixir is a quite capable language and I will be exploring it further in future posts. For now we\u0026rsquo;ll simply transform the equivalent function in Elixir to its native AST to see what it looks like. This is really quite easy in Elixir as it has a far more natural metaprogramming experience than F#. We just use quote(opts, block), where block is the expression we want to get the AST representation for. As an example let\u0026rsquo;s quote the Elixir equivalent of the F# Test module with the square function:\nquote do defmodule Test do def square(x) do x * x end end end This results in an Elixir AST:\n{:defmodule, [context: Elixir, import: Kernel], [{:__aliases__, [alias: false], [:Test]}, [do: {:def, [context: Elixir, import: Kernel], [{:square, [context: Elixir], [{:x, [], Elixir}]}, [do: {:*, [context: Elixir, import: Kernel], [{:x, [], Elixir}, {:x, [], Elixir}]}]]}]]} A Tale Of Two Trees # Let\u0026rsquo;s start defining a tree structure for our transformation, we are going to be transforming from an F# TAST straight to an Elixir AST as Elixir has capabilities to use the AST fairly easily, we don\u0026rsquo;t have to specifically work with the AST to get it back to code.\ntype ElixirAst = | Fragment of atom : string * metadata : (string * string ) list * args : Arguments | Nested of (string * Arguments) list and Arguments = | Binding of string | Bindings of list\u0026lt;string\u0026gt; | Arguments of list\u0026lt;ElixirAst\u0026gt; We use a recursive Discriminated Union that describes the Elixir AST. You should be able to see it\u0026rsquo;s made up of either a Fragment or a Nested both of those can contain Arguments that can in turn contain Binding, Bindings, or a list of Arguments that can also be Fragment or Nested \u0026hellip;\nIt\u0026rsquo;s quite a brain twister at first glance but it should hopefully make sense.\nTransforming The Trees # We are now going to write a pair of functions that will iterate over the F# TAST and produce an Elixir AST, its more or less a case of transforming the F# constructs to the Elixir equivalents.\nlet rec traverseExpr expr = match expr with | BasicPatterns.Call(_expr,functionMemberOrVal,_,_typeSig, expressions) -\u0026gt; let argumentFragments = expressions |\u0026gt; List.map traverseExpr let functionName = functionMap functionMemberOrVal.LogicalName Fragment(functionName, [\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;;\u0026#34;import:\u0026#34;, \u0026#34;Kernel\u0026#34;], Arguments argumentFragments) | BasicPatterns.Value(v) -\u0026gt; Fragment(\u0026#34;:\u0026#34; + v.DisplayName, [], Binding \u0026#34;Elixir\u0026#34;) | other -\u0026gt; failwithf \u0026#34;Not implmented: %A\u0026#34; other let rec traverse decl = match decl with | FSharpImplementationFileDeclaration.Entity(ent, declList) -\u0026gt; let arguments = [yield Fragment(\u0026#34;:__aliases__\u0026#34;, [\u0026#34;alias:\u0026#34;, \u0026#34;false\u0026#34;], Bindings[\u0026#34;:\u0026#34; + ent.DisplayName]) let declList = declList |\u0026gt; List.map traverse yield Nested[(\u0026#34;do:\u0026#34;, Arguments declList)] ] let moduledef = Fragment(\u0026#34;:defmodule\u0026#34;, [\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;; \u0026#34;import:\u0026#34;, \u0026#34;Kernel\u0026#34;], Arguments arguments ) moduledef | FSharpImplementationFileDeclaration -\u0026gt; failwith \u0026#34;Not implemented\u0026#34; | FSharpImplementationFileDeclaration.MemberOrFunctionOrValue(memb, curriedParameterGroups, expr)-\u0026gt; let args = [ let parameters = [for group in curriedParameterGroups do for param in group do yield \u0026#34;:\u0026#34; + param.DisplayName] |\u0026gt; List.map (fun name -\u0026gt; Fragment(name, [], Binding \u0026#34;Elixir\u0026#34;)) yield Fragment(\u0026#34;:\u0026#34; + memb.DisplayName, [\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;], Arguments parameters) yield Nested[(\u0026#34;do:\u0026#34;, Arguments [traverseExpr expr] ) ] ] Fragment(\u0026#34;:def\u0026#34;, [\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;; \u0026#34;import:\u0026#34;, \u0026#34;Kernel\u0026#34;], Arguments args) Notice the FSharpImplementationFileDeclaration fails with not implemented as it\u0026rsquo;s not needed in this example so we will omit it here.\nOne thing I haven\u0026rsquo;t shown in detail in the F# TAST section was the expressions, that\u0026rsquo;s because there are 43 different elements that make up F#\u0026rsquo;s\u0026rsquo; full typed expressions. In our example we only need to use Call and Value so the other expression types fail with not implemented too.\nLet\u0026rsquo;s try this out by pumping the F# TAST through our new function and see what we get:\nlet astElements = [for implFile in checkProjectResults.AssemblyContents.ImplementationFiles do for decl in implFile.Declarations do yield traverse decl] [Fragment (\u0026#34;:defmodule\u0026#34;,[(\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;); (\u0026#34;import:\u0026#34;, \u0026#34;Kernel\u0026#34;)], Arguments [Fragment (\u0026#34;:__aliases__\u0026#34;,[(\u0026#34;alias:\u0026#34;, \u0026#34;false\u0026#34;)],Bindings [\u0026#34;:Test\u0026#34;]); Nested [(\u0026#34;do:\u0026#34;, Arguments [Fragment (\u0026#34;:def\u0026#34;,[(\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;); (\u0026#34;import:\u0026#34;, \u0026#34;Kernel\u0026#34;)], Arguments [Fragment (\u0026#34;:square\u0026#34;,[(\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;)], Arguments [Fragment (\u0026#34;:x\u0026#34;,[],Binding \u0026#34;Elixir\u0026#34;)]); Nested [(\u0026#34;do:\u0026#34;, Arguments [Fragment (\u0026#34;:*\u0026#34;, [(\u0026#34;context:\u0026#34;, \u0026#34;Elixir\u0026#34;); (\u0026#34;import:\u0026#34;, \u0026#34;Kernel\u0026#34;)], Arguments [Fragment (\u0026#34;:x\u0026#34;,[],Binding \u0026#34;Elixir\u0026#34;); Fragment (\u0026#34;:x\u0026#34;,[],Binding \u0026#34;Elixir\u0026#34;)])])]])])]])] \u0026ldquo;I have harnessed the shadows that stride from world to world to sow death and madness.\u0026rdquo;\nStringify the AST # Ok, so now that we have an F# based tree structure that describes the Elixir AST we now need it back in stringly form to do anything with it. We could of just traversed the tree writing the AST elements but that wouldn\u0026rsquo;t of given us the flexibility to traverse the tree in the future.\nAs we will be working with string\u0026rsquo;s and StringBuilders let\u0026rsquo;s define a few helper functions:\nlet (++) (ctx:StringBuilder) (str:String) = ctx.Append(str) let (+\u0026gt;) (ctx:StringBuilder) (f:StringBuilder -\u0026gt; StringBuilder) = f ctx These will let us pipe two StringBuilder append operations together and also allow us to pipe into other functions that take a StringBuilder as an argument.\nFirst we\u0026rsquo;ll define something simple that will just render the metadata:\nlet printMeta (meta: (string * string) list) (sb:StringBuilder) = meta |\u0026gt; Seq.map (fun (key,value) -\u0026gt; sprintf \u0026#34;%s %s\u0026#34; key value) |\u0026gt; String.concat \u0026#34;, \u0026#34; |\u0026gt; Printf.bprintf sb \u0026#34;[%s]\u0026#34; sb Now a duo of functions that will traverse the ElixirAst, processing the Arguments, Fragment, and Nested elements:\nlet rec printArgs args (sb:StringBuilder) = match args with | Binding s -\u0026gt; sb ++ s | Bindings xs -\u0026gt; sb ++ sprintf \u0026#34;[%s]\u0026#34; (String.concat \u0026#34;, \u0026#34; xs) | Arguments args -\u0026gt; sb ++ \u0026#34;[\u0026#34; |\u0026gt; ignore args |\u0026gt; List.iteri (fun i item -\u0026gt; printAst item sb |\u0026gt; ignore if i \u0026lt; args.Length-1 then sb ++ \u0026#34;, \u0026#34; |\u0026gt; ignore) sb ++ \u0026#34;]\u0026#34; and printAst ast (sb:StringBuilder) = match ast with | Fragment(atom, metadata, args) -\u0026gt; sb ++ sprintf \u0026#34;{%s, \u0026#34; atom +\u0026gt; printMeta metadata ++ \u0026#34;, \u0026#34; +\u0026gt; printArgs args ++ \u0026#34;}\u0026#34; | Nested(items) -\u0026gt; sb ++ \u0026#34;[\u0026#34; |\u0026gt; ignore items |\u0026gt; List.iteri (fun i (name, args) -\u0026gt; sb ++ sprintf \u0026#34;%s \u0026#34; name +\u0026gt; printArgs args |\u0026gt; ignore if i \u0026lt; items.Length-1 then sb ++ \u0026#34;, \u0026#34; |\u0026gt; ignore) sb ++ \u0026#34;]\u0026#34; Finally capture the results by iterating over the AST and collecting the stringified results:\nlet elixirAstItems = let sb = StringBuilder() [for element in astElements do printAst element sb |\u0026gt; ignore yield sb.ToString()] VM execution # \u0026ldquo;I have brought to light a monstrous abnormality, but I did it for the sake of knowledge. Now for the sake of all life and Nature you must help me thrust it back into the dark again.\u0026rdquo;\nHelper Process # We\u0026rsquo;ll go for the simplest option here: we will create a new process, start up an interactive Elixir session, and send across a few commands including the AST that we just created.\nNow we\u0026rsquo;ll define a type to help us with this:\ntype public ProcessWrapper(file: string, env:Collections.Generic.IDictionary\u0026lt;_,_\u0026gt;) = let info = ProcessStartInfo(RedirectStandardInput = true, RedirectStandardOutput = true, UseShellExecute = false, CreateNoWindow = true, FileName = file) do for item in env do info.EnvironmentVariables.[item.Key] \u0026lt;- item.Value let proc = new Process(StartInfo = info) let outputBuffer = StringBuilder() let dataReceived = proc.OutputDataReceived.Subscribe(fun data -\u0026gt; outputBuffer.AppendLine(data.Data) |\u0026gt; ignore) [\u0026lt;CLIEvent\u0026gt;] member this.OutputReceived = proc.OutputDataReceived [\u0026lt;CLIEvent\u0026gt;] member this.ErrorReceived = proc.ErrorDataReceived member this.Start() = proc.Start() |\u0026gt; ignore proc.BeginOutputReadLine() member this.Send(line: string) = proc.StandardInput.WriteLine(line) member this.Flush() = proc.StandardInput.Flush() member x.GetOutputBuffer(?clear) = let data = outputBuffer.ToString() if clear.IsSome then outputBuffer.Clear() |\u0026gt; ignore data interface IDisposable with member x.Dispose() = dataReceived.Dispose() proc.Dispose() Code Execution # Now that we have all of the pieces in place let\u0026rsquo;s start the process and add some path variables so that iex can find the things that are normally in your path.\nlet iex = new ProcessWrapper(\u0026#34;/usr/local/Cellar/elixir/1.0.4/bin/iex\u0026#34;, dict[\u0026#34;PATH\u0026#34;, \u0026#34;/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin\u0026#34;]) iex.Start() Now we can send across out AST definition and assign it to mycode\niex.Send(\u0026#34;mycode =\u0026#34; + List.head elixirAstItems) If we examined the output of iex now we can see that this has been assigned:\niex.GetOutputBuffer() {:defmodule, [context: Elixir, import: Kernel], [{:__aliases__, [alias: false], [:Test]}, [do: [{:def, [context: Elixir, import: Kernel], [{:square, [context: Elixir], [{:x, [], Elixir}]}, [do: [{:*, [context: Elixir, import: Kernel], [{:x, [], Elixir}, {:x, [], Elixir}]}]]]}]]]} Now we can use Elixir\u0026rsquo;s metaprogramming capabilities to compile this AST so we can use it:\niex.Send(\u0026#34;mycode |\u0026gt; Code.compile_quoted()\u0026#34;) [{Test, \u0026lt;\u0026lt;70, 79, 82, 49, 0, 0, 4, 104, 66, 69, 65, 77, 69, 120, 68, 99, 0 ...\u0026gt;\u0026gt;}] Yes!! It looks like our AST has now been compiled, we have a module named Test with binary data assigned. Now let\u0026rsquo;s see if we can execute our square function:\niex.Send(\u0026#34;Test.square(16)\u0026#34;) [256] Woo-hoo job done! I hope you enjoyed this whistle stop tour through the Mountains Of Madness.\nI don\u0026rsquo;t know about you, but I need a cup of tea!\nUntil next time \u0026hellip;\n","date":"July 12, 2015","externalUrl":null,"permalink":"/programming/2015-07-08-meta-matic/","section":"Blog","summary":"Sit down, strap in, and prepare for take off, we’re going Meta-Matic!\n","title":"Meta-Matic","type":"programming"},{"content":"Did you know there was more to the type matching operator than just pattern matching and exception handling?\nThe type matching operator is defined thus: :? It can be used by pattern matching to match on a specific type. For example, you might want to test that an object is a certain type or deal with an object being one of several different types. Pattern matching on types is your friend here:\nmatch symbolUse.Symbol with | :? FSharpMemberOrFunctionOrValue | :? FSharpUnionCase | :? FSharpEntity | :? FSharpField | :? FSharpGenericParameter | :? FSharpActivePatternCase | :? FSharpParameter | :? FSharpStaticParameter -\u0026gt; match getSymbolDeclarationLocation symbolUse currentFile solution with | SymbolDeclarationLocation.External -\u0026gt; false | SymbolDeclarationLocation.Unknown -\u0026gt; false | _ -\u0026gt; true | _ -\u0026gt; false During pattern matching you can also use the as assignment operator to assign a named binding to the match so you can use it directly. This is somewhat akin to using is and as in C#, or using an as and then a null check. Yuck! None of that kind of thing in F#:\nlet isPrivateToFile = match symbolUse.Symbol with | :? FSharpMemberOrFunctionOrValue as m -\u0026gt; not m.IsModuleValueOrMember | :? FSharpEntity as m -\u0026gt; m.Accessibility.IsPrivate | :? FSharpGenericParameter -\u0026gt; true | :? FSharpUnionCase as m -\u0026gt; m.Accessibility.IsPrivate | :? FSharpField as m -\u0026gt; m.Accessibility.IsPrivate | _ -\u0026gt; false It can also be used in exception handing to match a specific type of exception, as in this example where TimeoutExceptions are caught:\nmember x.GetDeclarationSymbols(line, col, lineStr) = match infoOpt with | None -\u0026gt; None | Some (checkResults, parseResults) -\u0026gt; let longName,residue = Parsing.findLongIdentsAndResidue(col, lineStr) // Get items \u0026amp; generate output try let results = Async.RunSynchronously (checkResults.GetDeclarationListSymbols(Some parseResults, line, col, lineStr, longName, residue, fun _ -\u0026gt; false), timeout = ServiceSettings.blockingTimeout ) Some (results, residue) with :? TimeoutException -\u0026gt; None A final use for :? that people either don\u0026rsquo;t tend to use or know about, is during a normal expression assignment. In this example item :? DotNetProject would evaluate to true when item is a DotNetProject.\noverride x.SupportsItem(item:IBuildTarget) = item :? DotNetProject Although not used that often I find the :? operator to be really useful.\nAs usual F# helps to keep things short, succinct, and sweet!\nUntil next time!\n","date":"March 30, 2015","externalUrl":null,"permalink":"/programming/2015-03-30-are-you-my-type/","section":"Blog","summary":"Did you know there was more to the type matching operator than just pattern matching and exception handling?\n","title":"Are you my type?","type":"programming"},{"content":"","date":"February 5, 2015","externalUrl":null,"permalink":"/astronomy/","section":"Astronomies","summary":"","title":"Astronomies","type":"astronomy"},{"content":"After enjoying owning a 150mm refractor telescope for a year, I decided I wanted to upgrade.\nOne of the reasons was the optics, I wanted to reduce the amount of chromatic aberration present, I also wanted to have a better focuser. The focuser was a real source of frustration with it constantly causing the view to shift while focussing. I didn\u0026rsquo;t want to spend a great deal of money on a new focuser when I wasn\u0026rsquo;t happy with the overall telescope anyway.\nIn optics, chromatic aberration (CA, also called achromatism, chromatic distortion, and spherochromatism) is a type of distortion in which there is a failure of a lens to focus all colour\u0026rsquo;s to the same convergence point.\nI didn\u0026rsquo;t want to get a Catadioptric style telescopes or a large reflector due to the size and also the maintenance aspects of keeping the various mirrors and lenses in alignment, not to mention the size and weight. To put it simply, I wanted a a telescope that I could take outside (without a crane) and observe that was also low maintenance.\nI did a lot of research other contenders for me were the APM 115mm refractor and the Williams Optics 102mm refractor. What drew me to the Takahashi TSA 102s was there were a lots of really good reviews available and Takahashi are also renowned for using good quality optics. I believe the lens cell is manufactured my Cannon.\nAt the same time I also decided to upgrade my mount. I have my 150mm refractor mounted on a SkyWatcher EQ5 mount. While this was adequate when I started I feel I have outgrown it somewhat. It is not a driven mount so I wanted something with a drive system to keep the stars centered in the view, and I also wanted something with a goto capability.\nI finally decided, after much research, to purchase a Takahashi TSA 102s 102mm Triplet refractor with a 3 inch Starlight Instruments feathertouch focuser, along with an iOptron IEQ45 mount.\nHere they are together in all their glory:\nDetail of the iOptron IEQ45 mount and controller: Mount detail showing latitude and azimuth adjusters: Detail of the superb Takahashi triplet lens: All I need now is a clear night to try them out! First Light # A few night later the sky was clear. At sunset I could see Venus and Mars setting, it was too late to get the mount and telescope setup to look at Mars or Venus, but I did manage to get some nice views with binoculars.\nSetup # Setup was really easy, align the scope roughly North, then use the polar alignment feature on the hand controller. A clock face is shown on the controller of the direction that you need to align the telescope to. You simply have to rotate the adjustment knobs on the mount to get Polaris aligned in the illuminated reticle.\nDouble Stars # As a quick test I managed to split a few double stars. Notably Rigel, Mintaka, Nair al saif, Meissa, Castor, and Polaris. There were a few more I managed to split with my limited eye pieces but I neglected to keep a log of them!\nPlanets # Well, only Jupiter was visible, but as usual it was a magnificent sight, it was a joy to have the great planet automatically tracked and centered in the eyepiece. I could make out some lovely detail in the cloud bands.\nDeep Sky # My knowledge of deep sky is not amazing, but I did manage to get great views of Plaides, Orion Nebula, Andromeda Galaxy. Beehive cluster.\nOh, I was also lucky enough to see the comet C/2014 Q2, otherwise know as Lovejoy!\nConclusion # The Takahashi TSA 102S is built magnificently, the finish is flawless, the lens is superb, the focuser doesn\u0026rsquo;t move the view out of alignment and is super smooth to use. The stars are crystal clear and pin sharp!\nThe iOptron IEQ45 is engineered really well too, the polar alignment is a joy to use with the illuminated view finder and helper function from the controller. The tracking and moving the mount with the controller works really well too.\nWhats Next # Well now my main optics and mount are in order I need some proper eyepieces. I only have the stock eyepieces that came with my 150mm: a 10mm and 25mm plössl. I seriously need to upgrade them pronto. I have my eye on nice TeleVue Delos eyepiece.\nNow that I have a stable mount I want to try some astrophotography, it would of been great showing you what I saw rather than just describing it.\nUntil next time!\n","date":"February 5, 2015","externalUrl":null,"permalink":"/astronomy/2015-02-05-dioptric-dilemmas-of-the-third-kind/","section":"Astronomies","summary":"After enjoying owning a 150mm refractor telescope for a year, I decided I wanted to upgrade.\n","title":"Dioptric Dilemmas Of The Third Kind","type":"post"},{"content":"","date":"December 10, 2014","externalUrl":null,"permalink":"/tags/monodevelop/","section":"Tags","summary":"","title":"Monodevelop","type":"tags"},{"content":"So this is my Christmas special. I\u0026rsquo;ve been asked on numerous times to write about the F# addin for Xamarin studio which is in the fsharpbinding repo, this repo is shared with the emacs support and also the Sublime Text support. So in this edition we will be taking a deep dive into the terrifying deep depths of the F# compiler and F# addin development\u0026hellip;\nActually I\u0026rsquo;m only joking, adding features to the F# compiler and F# addin is fairly easy depending on what you want to do. You can run into issues along the way which means you might need to delve into more of the F# compilers functionality, essentially to derive and adapt new functions that you might need.\nWhat I\u0026rsquo;m going to show is how to add a new autocompletion list where instead of a standard completion list, its categorised by the type that the methods are derived from. As an example you would be able to see the ToString methods etc on the Obj type andy other methods defined on their particular derived type.\nOne door leads to the source # Lets have a look at the current completion list function in the F# compiler:\nmember GetDeclarationsAlternate : ParsedFileResultsOpt:ParseFileResults option * line: int * colAtEndOfPartialName: int * lineText:string * qualifyingNames: string list * partialName: string * ?hasTextChangedSinceLastTypecheck: (obj * range -\u0026gt; bool) -\u0026gt; Async\u0026lt;DeclarationSet\u0026gt; Essentially this function takes a lot of parameters, I don\u0026rsquo;t want to go into the details too much as the FCS sample pages does a good job of that. So what is a DeclarationSet? Well as you expect its a collection of Declarations. A Declaration has a Glyph, Name, and DescriptionText. The DescriptionText is a ToolTipText which is a text based rendering of the declaration in question.\nToolTipElement # /// Describe a comment as either a block of text or a file+signature reference into an intellidoc file. type XmlComment = | XmlCommentNone | XmlCommentText of string | XmlCommentSignature of (*File and Signature*) string * string /// A single data tip display element type ToolTipElement = | ToolTipElementNone /// A single type, method, etc with comment. | ToolTipElement of (* text *) string * XmlComment // /// A parameter of a method. // | ToolTipElementParameter of string * XmlComment * string /// For example, a method overload group. | ToolTipElementGroup of ((* text *) string * XmlComment) list /// An error occurred formatting this element | ToolTipElementCompositionError of string At first glance this information is quite interesting but in use the limitation of text based rendering becomes apparent. How can you break down the information into easily renderable parts or know the underlying types that make up the declaration. Text based manipulation means a lot of work, and also lots of potential bugs, as you would expect with text based or weakly typed system.\nLets have a look at the GetDeclarationsAlternate function and see if we have access to any detailed information:\nGetDeclarationsAlternate # member info.GetDeclarationsAlternate(parseResultsOpt, line, colAtEndOfNamesAndResidue, lineStr, qualifyingNames, partialName, ?hasTextChangedSinceLastTypecheck) = let hasTextChangedSinceLastTypecheck = defaultArg hasTextChangedSinceLastTypecheck (fun _ -\u0026gt; false) reactorOp DeclarationSet.Empty (fun scope -\u0026gt; scope.GetDeclarations(parseResultsOpt, line, lineStr, colAtEndOfNamesAndResidue, qualifyingNames, partialName, hasTextChangedSinceLastTypecheck)) OK, so that just calls GetDeclarations after doing a check for changes since the last type check, lets go deeper\u0026hellip;\nGetDeclarations # member x.GetDeclarations (parseResultsOpt:ParseFileResults option, line, lineStr, colAtEndOfNamesAndResidue, qualifyingNames, partialName, hasTextChangedSinceLastTypecheck) : DeclarationSet = let isInterfaceFile = SourceFileImpl.IsInterfaceFile mainInputFileName ErrorScope.Protect Range.range0 (fun () -\u0026gt; match GetDeclItemsForNamesAtPosition(parseResultsOpt, Some qualifyingNames, Some partialName, line, lineStr, colAtEndOfNamesAndResidue, ResolveTypeNamesToCtors, ResolveOverloads.Yes, hasTextChangedSinceLastTypecheck) with | None -\u0026gt; DeclarationSet.Empty | Some(items,denv,m) -\u0026gt; let items = items |\u0026gt; filterIntellisenseCompletionsBasedOnParseContext (parseResultsOpt |\u0026gt; Option.bind (fun x -\u0026gt; x.ParseTree)) (mkPos line colAtEndOfNamesAndResidue) let items = if isInterfaceFile then items |\u0026gt; List.filter IsValidSignatureFileItem else items DeclarationSet.Create(infoReader,m,denv,items,reactorOps,checkAlive)) (fun msg -\u0026gt; DeclarationSet.Error msg) Right, this is more interesting, if we look at the pattern match match filterIntellisenseCompletionsBasedOnParseContext you can see we have there are items, denv, and m. Now what exactly is an Item?\nLets go deeper still and take a look\u0026hellip;.\nItem # /// Represents an item that results from name resolution type Item = /// Represents the resolution of a name to an F# value or function. | Value of ValRef /// Represents the resolution of a name to an F# union case. | UnionCase of UnionCaseInfo /// Represents the resolution of a name to an F# active pattern result. | ActivePatternResult of ActivePatternInfo * TType * int * range /// Represents the resolution of a name to an F# active pattern case within the body of an active pattern. | ActivePatternCase of ActivePatternElemRef /// Represents the resolution of a name to an F# exception definition. | ExnCase of TyconRef /// Represents the resolution of a name to an F# record field. | RecdField of RecdFieldInfo // The following are never in the items table but are valid results of binding an identitifer in different circumstances. /// Represents the resolution of a name at the point of its own definition. | NewDef of Ident /// Represents the resolution of a name to a .NET field | ILField of ILFieldInfo /// Represents the resolution of a name to an event | Event of EventInfo /// Represents the resolution of a name to a property | Property of string * PropInfo list /// Represents the resolution of a name to a group of methods | MethodGroup of string * MethInfo list /// Represents the resolution of a name to a constructor | CtorGroup of string * MethInfo list /// Represents the resolution of a name to the fake constructor simulated for an interface type. | FakeInterfaceCtor of TType /// Represents the resolution of a name to a delegate | DelegateCtor of TType /// Represents the resolution of a name to a group of types | Types of string * TType list /// CustomOperation(nm, helpText, methInfo) /// Used to indicate the availability or resolution of a custom query operation such as \u0026#39;sortBy\u0026#39; or \u0026#39;where\u0026#39; in computation expression syntax | CustomOperation of string * (unit -\u0026gt; string option) * MethInfo option /// Represents the resolution of a name to a custom builder in the F# computation expression syntax | CustomBuilder of string * ValRef /// Represents the resolution of a name to a type variable | TypeVar of string * Typar /// Represents the resolution of a name to a module or namespace | ModuleOrNamespaces of Tast.ModuleOrNamespaceRef list /// Represents the resolution of a name to an operator | ImplicitOp of Ident * TraitConstraintSln option ref /// Represents the resolution of a name to a named argument | ArgName of Ident * TType * ArgumentContainer option /// Represents the resolution of a name to a named property setter | SetterArg of Ident * Item /// Represents the potential resolution of an unqualified name to a type. | UnqualifiedType of TyconRef list Finally lets look at the Create function of DeclarationSet to see what\u0026rsquo;s involved:\nDeclarationSet - Create # // Make a \u0026#39;Declarations\u0026#39; object for a set of selected items static member Create(infoReader:InfoReader, m, denv, items, reactor, checkAlive) = let g = infoReader.g let items = items |\u0026gt; RemoveExplicitlySuppressed g // Sort by name. For things with the same name, // - show types with fewer generic parameters first // - show types before over other related items - they usually have very useful XmlDocs let items = items |\u0026gt; List.sortBy (fun d -\u0026gt; let n = match d with | Item.Types (_,(TType_app(tcref,_) :: _)) -\u0026gt; 1 + tcref.TyparsNoRange.Length // Put delegate ctors after types, sorted by #typars. RemoveDuplicateItems will remove FakeInterfaceCtor and DelegateCtor if an earlier type is also reported with this name | Item.FakeInterfaceCtor (TType_app(tcref,_)) | Item.DelegateCtor (TType_app(tcref,_)) -\u0026gt; 1000 + tcref.TyparsNoRange.Length // Put type ctors after types, sorted by #typars. RemoveDuplicateItems will remove DefaultStructCtors if a type is also reported with this name | Item.CtorGroup (_, (cinfo :: _)) -\u0026gt; 1000 + 10 * (tcrefOfAppTy g cinfo.EnclosingType).TyparsNoRange.Length | _ -\u0026gt; 0 (d.DisplayName,n)) // Remove all duplicates. We\u0026#39;ve put the types first, so this removes the DelegateCtor and DefaultStructCtor\u0026#39;s. let items = items |\u0026gt; RemoveDuplicateItems g if verbose then dprintf \u0026#34;service.ml: mkDecls: %d found groups after filtering\\n\u0026#34; (List.length items); // Group by display name let items = items |\u0026gt; List.groupBy (fun d -\u0026gt; d.DisplayName) // Filter out operators (and list) let items = // Check whether this item looks like an operator. let isOpItem(nm,item) = match item with | [Item.Value _] | [Item.MethodGroup(_,[_])] -\u0026gt; (IsOpName nm) \u0026amp;\u0026amp; nm.[0]=\u0026#39;(\u0026#39; \u0026amp;\u0026amp; nm.[nm.Length-1]=\u0026#39;)\u0026#39; | [Item.UnionCase _] -\u0026gt; IsOpName nm | _ -\u0026gt; false let isFSharpList nm = (nm = \u0026#34;[]\u0026#34;) // list shows up as a Type and a UnionCase, only such entity with a symbolic name, but want to filter out of intellisense items |\u0026gt; List.filter (fun (nm,items) -\u0026gt; not (isOpItem(nm,items)) \u0026amp;\u0026amp; not(isFSharpList nm)) let decls = // Filter out duplicate names items |\u0026gt; List.map (fun (nm,itemsWithSameName) -\u0026gt; match itemsWithSameName with | [] -\u0026gt; failwith \u0026#34;Unexpected empty bag\u0026#34; | items -\u0026gt; new Declaration(nm, GlyphOfItem(denv,items.Head), Choice1Of2 (items, infoReader, m, denv, reactor, checkAlive))) new DeclarationSet(Array.ofList decls) This looks very promising, this information could be just what we need. If we do a quick search and see what else uses Items so we can get a better idea of how its used. Lets just see if there are any pattern matches for Item.Value to get a quick idea:\nnameres.fs tc.fs fsi.fs service.fs ServiceDeclarations.fs Symbols.fs You\u0026rsquo;re a symbol for your kind # The matches in Symbol.fs look interesting, you can see it\u0026rsquo;s relatively easy to construct a Symbol if you have access to the relevant parts. Having a list of symbols available rather than a DeclarationSet of ToolTipElement could be just what we need.\nLets look at constructing a symbol rather than the declaration set:\nmember x.GetDeclarationListSymbols (parseResultsOpt:FSharpParseFileResults option, line, lineStr, colAtEndOfNamesAndResidue, qualifyingNames, partialName, hasTextChangedSinceLastTypecheck) = let isInterfaceFile = SourceFileImpl.IsInterfaceFile mainInputFileName ErrorScope.Protect Range.range0 (fun () -\u0026gt; match GetDeclItemsForNamesAtPosition(parseResultsOpt, Some qualifyingNames, Some partialName, line, lineStr, colAtEndOfNamesAndResidue, ResolveTypeNamesToCtors, ResolveOverloads.Yes, hasTextChangedSinceLastTypecheck) with | None -\u0026gt; List.Empty | Some(items,_denv,_m) -\u0026gt; let items = items |\u0026gt; filterIntellisenseCompletionsBasedOnParseContext (parseResultsOpt |\u0026gt; Option.bind (fun x -\u0026gt; x.ParseTree)) (mkPos line colAtEndOfNamesAndResidue) let items = if isInterfaceFile then items |\u0026gt; List.filter IsValidSignatureFileItem else items //do filtering like Declarationset let items = items |\u0026gt; RemoveExplicitlySuppressed g // Sort by name. For things with the same name, // - show types with fewer generic parameters first // - show types before over other related items - they usually have very useful XmlDocs let items = items |\u0026gt; List.sortBy (fun d -\u0026gt; let n = match d with | Item.Types (_,(TType_app(tcref,_) :: _)) -\u0026gt; 1 + tcref.TyparsNoRange.Length // Put delegate ctors after types, sorted by #typars. RemoveDuplicateItems will remove FakeInterfaceCtor and DelegateCtor if an earlier type is also reported with this name | Item.FakeInterfaceCtor (TType_app(tcref,_)) | Item.DelegateCtor (TType_app(tcref,_)) -\u0026gt; 1000 + tcref.TyparsNoRange.Length // Put type ctors after types, sorted by #typars. RemoveDuplicateItems will remove DefaultStructCtors if a type is also reported with this name | Item.CtorGroup (_, (cinfo :: _)) -\u0026gt; 1000 + 10 * (tcrefOfAppTy g cinfo.EnclosingType).TyparsNoRange.Length | _ -\u0026gt; 0 (d.DisplayName,n)) // Remove all duplicates. We\u0026#39;ve put the types first, so this removes the DelegateCtor and DefaultStructCtor\u0026#39;s. let items = items |\u0026gt; RemoveDuplicateItems g if verbose then dprintf \u0026#34;service.ml: mkDecls: %d found groups after filtering\\n\u0026#34; (List.length items); // Group by display name let items = items |\u0026gt; List.groupBy (fun d -\u0026gt; d.DisplayName) // Filter out operators (and list) let items = // Check whether this item looks like an operator. let isOpItem(nm,item) = match item with | [Item.Value _] | [Item.MethodGroup(_,[_])] -\u0026gt; (IsOpName nm) \u0026amp;\u0026amp; nm.[0]=\u0026#39;(\u0026#39; \u0026amp;\u0026amp; nm.[nm.Length-1]=\u0026#39;)\u0026#39; | [Item.UnionCase _] -\u0026gt; IsOpName nm | _ -\u0026gt; false let isFSharpList nm = (nm = \u0026#34;[]\u0026#34;) // list shows up as a Type and a UnionCase, only such entity with a symbolic name, but want to filter out of intellisense items |\u0026gt; List.filter (fun (nm,items) -\u0026gt; not (isOpItem(nm,items)) \u0026amp;\u0026amp; not(isFSharpList nm)) let items = // Filter out duplicate names items |\u0026gt; List.map (fun (_nm,itemsWithSameName) -\u0026gt; match itemsWithSameName with | [] -\u0026gt; failwith \u0026#34;Unexpected empty bag\u0026#34; | items -\u0026gt; items |\u0026gt; List.map (fun item -\u0026gt; let symbol = FSharpSymbol.Create(g, thisCcu, tcImports, item) FSharpSymbolUse(g, _denv, symbol, ItemOccurence.Use, _m))) //end filtering items) (fun _msg -\u0026gt; []) Looking at the code you can see its various pieces cobbled together to construct an FSharpSymbolUSe rather than a DeclarationSet. This should allow us to create a more elaborate autocompletion which displays members by base type rather than a flat list.\nThere Are No Flowers in the Real World… # So that\u0026rsquo;s the easy bit done, now over to MonoDevelop. We need to rip out the old completions and splice in the new one, currently it\u0026rsquo;s defined in FSharpTextEditorCompletion and FSharpMemberCompletionData.\nLets have a look at CompletionData which we will need to recreate for our purposes:\nCompletionData # type CompletionData abstract member Icon : IconId with get, set abstract member DisplayText : string with get, set abstract member Description : string with get, set abstract member CompletionText : string with get, set abstract member GetDisplayDescription : bool -\u0026gt; string abstract member GetRightSideDescription : bool -\u0026gt; string abstract member CompletionCategory : CompletionCategory with get, set abstract member DisplayFlags : DisplayFlags with get, set abstract member CreateTooltipInformation : bool -\u0026gt; TooltipInformation abstract member HasOverloads : () -\u0026gt; bool abstract member OverloadedData : () -\u0026gt; IEnumerable\u0026lt;ICompletionData\u0026gt; abstract member AddOverload : ICompletionData -\u0026gt; () abstract member InsertCompletionText : CompletionListWindow * ref KeyActions * Gdk.Key * char * Gdk.ModifierType -\u0026gt; () abstract member CompareTo : obj -\u0026gt; int All of these are virtual in the CompletionData type, what we will need to do is add overrides for the HasOverloads, OverloadedData, AddOverload, and CreateTooltipInformation to give us the functionality we require. It\u0026rsquo;s going to be vety similar to the old code except we will be using symbols rather than ToolTipElement data to create the completion data.\nLets create a new FSharpMemberCompletionData:\ntype internal FSharpMemberCompletionDataSorted(name, icon, symbol:FSharpSymbol, overloads:FSharpSymbol seq) = inherit CompletionData(CompletionText = Lexhelp.Keywords.QuoteIdentifierIfNeeded name, DisplayText = name, DisplayFlags = DisplayFlags.DescriptionHasMarkup, Icon = icon) /// Check if the datatip has multiple overloads override x.HasOverloads = not (Seq.isEmpty overloads) /// Split apart the elements into separate overloads override x.OverloadedData = overloads |\u0026gt; Seq.map (fun symbol -\u0026gt; FSharpMemberCompletionDataSorted(symbol.DisplayName, icon, symbol, Seq.empty) :\u0026gt; _ ) override x.AddOverload (data: ICompletionData) = () override x.CreateTooltipInformation (smartWrap: bool) = let tip = SymbolTooltips.getTooltipFromSymbol symbol FSharpDisplayContext.Empty None match tip with | ToolTips.ToolTip (signature, xmldoc) -\u0026gt; let toolTipInfo = new TooltipInformation(SignatureMarkup = signature) match xmldoc with | Full(summary) -\u0026gt; toolTipInfo.SummaryMarkup \u0026lt;- summary toolTipInfo | Lookup(key, potentialFilename) -\u0026gt; let summary = maybe {let! filename = potentialFilename let! markup = TipFormatter.findDocForEntity(filename, key) let summary = Tooltips.getTooltip Styles.simpleMarkup markup return summary } summary |\u0026gt; Option.iter (fun summary -\u0026gt; toolTipInfo.SummaryMarkup \u0026lt;- summary) toolTipInfo | EmptyDoc -\u0026gt; toolTipInfo | _ -\u0026gt; TooltipInformation() In this section you can see the use of the maybe computation expression (You wont find the \u0026lsquo;M\u0026rsquo; word mentioned here thank you very much!) to simplify the creation of the Lookup tooltip\u0026rsquo;s. Lookup means pulling the information from monodoc which loads the xmldoc files, and Full means there is xmldoc\u0026rsquo;s present in the compiler. Full will occur in your own files and Lookup will occur in referenced assemblies.\nWe also need a define little type to hold the category as the CompletionCategory type is abstract:\ntype Category(category) = inherit CompletionCategory(category, null) override x.CompareTo other = compare x.DisplayText other.DisplayText Next we will add a function called getCompletionData to the existing FSharpTextEditorCompletion.\nlet getCompletionData (symbols:FSharpSymbol list list) = let categories = Dictionary\u0026lt;string, Category\u0026gt;() let getOrAddCategory id = let found, item = categories.TryGetValue id if found then item else let cat = Category id categories.Add (id,cat) cat let (|Function|Val|Unknown|) (symbol:FSharpSymbol) = match symbol with | MemberOrFunctionOrValue symbol when not (isConstructor symbol) -\u0026gt; if symbol.FullType.IsFunctionType \u0026amp;\u0026amp; not symbol.IsPropertyGetterMethod \u0026amp;\u0026amp; not symbol.IsPropertySetterMethod then Function symbol else Val symbol | _ -\u0026gt; Unknown symbol let symbolToIcon (s:FSharpSymbol) = match s with | ActivePatternCase _ -\u0026gt; Stock.Enum | Field _ -\u0026gt; Stock.Field | UnionCase _ -\u0026gt; Stock.Enum | Class -\u0026gt; Stock.Class | Delegate -\u0026gt; Stock.Delegate | Event -\u0026gt; Stock.Event | Property -\u0026gt; Stock.Property | Function _ -\u0026gt; MStock.Method | Val _ -\u0026gt; Stock.Field | Enum -\u0026gt; Stock.Enum | Interface -\u0026gt; Stock.Interface | Module -\u0026gt; Stock.Class | Namespace -\u0026gt; Stock.NameSpace | Record -\u0026gt; Stock.Class | Union -\u0026gt; Stock.Enum | ValueType -\u0026gt; Stock.Struct | _ -\u0026gt; Stock.Struct let symbolToCompletionData (symbol:FSharpSymbol) = let cd = FSharpMemberCompletionDataSorted(symbol.Head.DisplayName, symbolToIcon symbol.Head, symbol.Head, symbol.Tail) match symbol.Head with | :? FSharpMemberOrFunctionOrValue as func -\u0026gt; d.CompletionCategory \u0026lt;- getOrAddCategory func.EnclosingEntity.FullName | other -\u0026gt; cd.CompletionCategory \u0026lt;- getOrAddCategory (other.FullName.Substring (0, other.FullName.LastIndexOf \u0026#39;.\u0026#39;)) symbols |\u0026gt; List.map symbolToCompletionData :\u0026gt; ICompletionData) We have a few helper function\u0026rsquo;s here, getOrAddCategory to get or add categories. An active pattern (|Function|Val|Unknown|) to help to split MemberOrFunctionOrValue into Function, Val or Unknown sub types. symbolToIcon to get a stock icon to represent the different types of item that will appear in the completion list. And finally we have a map function, symbolToCompletionData which uses all of the other helper functions to project each symbol into a new FSharpMemberCompletionDataSorted. This is done by using either func.EnclosingEntity.FullName if the type match is FSharpMemberOrFunctionOrValue or other.FullName.Substring (0, other.FullName.LastIndexOf '.') if the type match is anything else.\nYou can see that the symbols are mapped using List.map and symbolToCompletionData at the end of the function. The resulting FSharpMemberCompletionDataSorted is finally coerced into an ICompletionData with the :\u0026gt; operator.\nFinally all that\u0026rsquo;s left is to change x.CodeCompletionCommandImpl in FSharpTextEditorCompletion, all we need to do is change the match statement to use the functions we defined above:\nmatch tyRes.GetDeclarations(line, col, lineStr) with | Some(decls, residue) when decls.Items.Any() -\u0026gt; let items = decls.Items |\u0026gt; Array.map (fun mi -\u0026gt; FSharpMemberCompletionData(mi) :\u0026gt; ICompletionData) result.AddRange(items) | _ -\u0026gt; () To use the new GetDeclarationSymbols function:\nmatch tyRes.GetDeclarationSymbols(line, col, lineStr) with | Some (symbols, residue) -\u0026gt; result.AddRange (getCompletionData symbols) | None -\u0026gt; () Phew! I think we are done. Spinning up Xamarin Studio with the new addin shows the new completion list:\nWe now have completion list sorted by the inheritor, which is especially nice for displaying members on hierarchical API\u0026rsquo;s. As a little bonus pressing Shift Up/Down will also move between the categories.\nSee, that wasn\u0026rsquo;t so scary was it?\nUntil next time!\n","date":"December 10, 2014","externalUrl":null,"permalink":"/programming/2014-12-07-terror-from-the-deep/","section":"Blog","summary":"So this is my Christmas special. I’ve been asked on numerous times to write about the F# addin for Xamarin studio which is in the fsharpbinding repo, this repo is shared with the emacs support and also the Sublime Text support. So in this edition we will be taking a deep dive into the terrifying deep depths of the F# compiler and F# addin development…\n","title":"Terror From The Deep","type":"programming"},{"content":"While I was visiting Boston earlier in the year I had the misfortune of kicking myself in the teeth with reflection. It\u0026rsquo;s something all programmers inevitably go through with reflection API\u0026rsquo;s as they are inherently untyped, a simple typo can leave you tearing out your hair or punching through your monitor! Yeah there\u0026rsquo;s things the horizon that will help namely the nameof expression in C#6 which should help in some areas, that\u0026rsquo;s if your willing to pay the price of using C#, but I wont go into that here :-). In F# we can leverage Type Providers fairly easily to wrap API usages in cases that we are interested in, or even create a general usage with a little more effort.\nUsing the Type Provider # In usage it will look like this vs the usual reflection API:\n//traditional reflection using untyped method let tt = typeof\u0026lt;DateTime\u0026gt; let meth = tt.GetMethod(\u0026#34;Add\u0026#34;) let result = meth.Invoke(DateTime.Now, [|TimeSpan.FromDays(1.)|]) //using the type provider to provide a little safety net type rt = TypedReflection.Reflection\u0026lt; \u0026#34;System.DateTime\u0026#34;, \u0026#34;AddSeconds\u0026#34;\u0026gt; let result = rt.AddSeconds(DateTime.Now, 1.) If you make a mistake the compiler will tell you and you will be forces to fix the typo or add namespace prefixes etc. You also get intellisense.autocompletion on usage and you can give actual parameters rather than arrays of loose objects etc.\nCode Dump # First of all I\u0026rsquo;m just going to leave the code here, and then talk through it below:\n[\u0026lt;TypeProvider\u0026gt;] type public ReflectionTypeProvider(config : TypeProviderConfig) as this = inherit TypeProviderForNamespaces() let assembly = Assembly.GetExecutingAssembly() let nameSpace = this.GetType().Namespace let providerType = ProvidedTypeDefinition(assembly, nameSpace, \u0026#34;Reflection\u0026#34;, Some typeof\u0026lt;obj\u0026gt;, IsErased = true, HideObjectMethods = true) let buildReflection typeName (parameters : obj[]) = let reflectionType = string parameters.[0] let methodName = string parameters.[1] let theType = Type.GetType(reflectionType, true) let meth = theType.GetMethod(methodName) if meth = null then failwith \u0026#34;No such method!\u0026#34; let wrapper = ProvidedTypeDefinition(assembly, nameSpace, typeName, Some (typeof\u0026lt;obj\u0026gt;), HideObjectMethods = true ) let parameterInfoToProvidedParameter (meth:MethodInfo) = let pi = meth.GetParameters() let instance = ProvidedParameter(\u0026#34;instance\u0026#34;, meth.ReflectedType) let parameters = pi |\u0026gt; Seq.map (fun p -\u0026gt; ProvidedParameter(p.Name, p.ParameterType) ) |\u0026gt; Seq.toList instance :: parameters let reflectionWrapper = ProvidedMethod (meth.Name, parameterInfoToProvidedParameter meth, meth.ReturnType, IsStaticMethod = true, InvokeCode = function | instance :: parameters -\u0026gt; try Expr.Call (instance, meth, parameters) with exn -\u0026gt; failwith \u0026#34;Error creating Invoke code.\u0026#34; | _ -\u0026gt; failwith \u0026#34;Error: unexpected number of parameters\u0026#34; ) wrapper.AddMember reflectionWrapper wrapper do providerType.DefineStaticParameters ([ ProvidedStaticParameter(\u0026#34;Type\u0026#34;, typeof\u0026lt;string\u0026gt;) ProvidedStaticParameter(\u0026#34;Method\u0026#34;, typeof\u0026lt;string\u0026gt;) ], buildReflection) this.AddNamespace (nameSpace, [ providerType ]) [\u0026lt;assembly:TypeProviderAssembly\u0026gt;] do() Skeleton code # Reading from the bottom up you can see the parameters that our Type Provider accepts are Type and Method, those a pretty self explanatory. You should also notice other boiler plate Type Provider code if you read my last ZipProvider post. The important part here is the buildReflection function.\nbuildReflection # First of all on lines 12/13 we scrape of the configuration parameters theType and meth, we then do a quick check to ensure the type and method actually exist, if they don\u0026rsquo;t we raise an error on line 17 so the use can correct the code.\nNext we create a variable named wrapper which wraps round the reflection API by creating a ProvidedTypeDefinition on line 19. We now have two methods which we use to create our safe API, parameterInfoToProvidedParameter and reflectionWrapper.\nparameterInfoToProvidedParameter # The purpose of this is a mapping function from the reflection API\u0026rsquo;s untyped abstract form to our typed form that we use in the construction of the Provided methods. Essentially this is pretty simple, we get the parameters for the MethodInfo which we are wrapping on line 23. The first parameter will be the instance of the reflected method will be working on, and the rest of the parameters will be those of the reflected method. To add those we loop over the parameters from the MethodInfo and map then to ProvidedProperties by using the Name and ParameterType properties.\n(Thinking about this we could do it slightly differently by adding a ProvidedConstructor which could take the initial instance, this could be added fairly easily if we really needed it. )\nreflectionWrapper # The reflectionWrapper is where the magic happens, we create a ProvidedMethod using the MethodInfo\u0026rsquo;s name\u0026rsquo;, we add the parameters by using the parameterInfoToProvidedParameter function, and we also add the return type by using the MethodInfo\u0026rsquo;s ReturnType parameter\u0026rsquo;. We can also take advantage of object initializers here to set IsStaticMethod to true, and to add in the invoke code.\nThe invoke code uses the function keyword which is really just a pattern match expression using only a single argument, here we use pattern matching on a list to extract the head|tail arguments. If you remember the parameterInfoToProvidedParameter function then you will know that it returns a list instance :: parameters. We can now use the Quotations Expr type with the Call function and pass in our instance and parameters in directly (instance is the reflected methods instance type, meth is the MethodInfo we will be calling, parameters are the parameters the MethodInfo requires.\nWrapping up # Finally we just add the ProvidedMethod reflectionWrapper to the ProvidedType wrapper\nThis is a fairly simple implementation but it could be beefed up quite easily into something a little more elaborate without too much trouble. If you use your imagination then there are numerous possibilities with Type Providers!\nReminds me of an old proverb:\nIf you have a problem ... if no one else can help ... and if you cant find an existing one ... maybe you can build ... a Type Provider. :-)\nUntil next time!\n","date":"November 16, 2014","externalUrl":null,"permalink":"/programming/2014-11-17-i-saw-my-reflection-and-cried-dot-dot-dot/","section":"Blog","summary":"While I was visiting Boston earlier in the year I had the misfortune of kicking myself in the teeth with reflection. It’s something all programmers inevitably go through with reflection API’s as they are inherently untyped, a simple typo can leave you tearing out your hair or punching through your monitor! Yeah there’s things the horizon that will help namely the nameof expression in C#6 which should help in some areas, that’s if your willing to pay the price of using C#, but I wont go into that here :-). In F# we can leverage Type Providers fairly easily to wrap API usages in cases that we are interested in, or even create a general usage with a little more effort.\n","title":"I saw my reflection and cried ...","type":"programming"},{"content":"First of all the title, redux because I\u0026rsquo;m revising post I started on earlier in the year, compression because this has to do with compression, and Flux, which is also part of the redux, one of the first things I remember writing on the net was an article about Flux Compression Generators on H2G2, its still there too!\nThis was a post I started writing back in January that I never got round to finishing.\nOnce upon a time I had a need to quickly browse a zip file and it\u0026rsquo;s Crc, so I quickly put together a Type Provider as a way to help in this en-devour. I\u0026rsquo;m going to split the code into a few section and run a commentary over each block so you can see what I did and why.\nZip Provider # I\u0026rsquo;m going to use SharpCompress as the basis for peering into zip files, you could also choose any other zip API. Essentially to open and peruse a zip the API consists of the following:\nlet zipfile = SharpCompress.Archive.ArchiveFactory.Open(fileName) for entry in zipFile.Entries do ... This gives us the ability to open a zip file and to iterate over its contents via a sequence of IArchiveEntry\nCreating the Type Provider # To create a Type Provider we need to create a type which looks like this, also notice the TypeProviderAssembly attribute:\n[\u0026lt;TypeProvider\u0026gt;] type public ZipProvider(cfg : TypeProviderConfig) as this = inherit TypeProviderForNamespaces() let asm = Assembly.GetExecutingAssembly() let ns = \u0026#34;Xebec\u0026#34; let root = ProvidedTypeDefinition(asm, ns, \u0026#34;ZipProvider\u0026#34;, Some(typeof\u0026lt;obj\u0026gt;)) let filePathParam = ProvidedStaticParameter(\u0026#34;FilePath\u0026#34;, typeof\u0026lt;string\u0026gt;) let buildTypes (typeName:string) (args:obj[]) = let fileName = args.[0] :?\u0026gt; string ... do root.DefineStaticParameters ([filePathParam], buildTypes) do this.AddNamespace(ns, [root]) [\u0026lt;TypeProviderAssembly\u0026gt;] do() Assuming buildTypes is complete and working the following user code might be used to use the Type Provider:\ntype myZip = Xebex.ZipProvider\u0026lt;\u0026#34;myfile.zip\u0026#34;\u0026gt; let file1Crc = myZip.MyFile1.Crc let file1Size = myZip.MyFile1.Size I always try think how the Type Provider might be used before undertaking work like this, Type Providers are supposed to aid usability not hinder it. I\u0026rsquo;m not a fan of big mechanistic design sessions and pencil pushing, I like to get into the field and working things out, that\u0026rsquo;s just my way though.\nBuild it and they will come # Next we need to create types based on the output of SharpCompress. The property zipFile.Entries returns a sequence of IArchiveEntry which have properties such as Size, Crc, FileName etc, so we\u0026rsquo;ll use these as we construct the type system.\nOne thing to be aware of with SharpCompress is the Entries properties returns a flat list of all the files in the archive. If you have a simple archive with only files at the root level then things are very simple. Once you move to an archive that has a complex directory hierarchy then things get a little trickier. One of the reasons is type namespace collisions, if we have file\u0026rsquo;s with the same name but different directories then the type system needs to match this to avoid adding a type with the same name. It doesn\u0026rsquo;t really make sense to have a flattened list anyway as I was using this provider to quickly peruse zip files from FSI.\nHere\u0026rsquo;s the bulk of buildTypes:\nlet buildTypes (typeName:string) (args:obj[]) = let fileName = args.[0] :?\u0026gt; string let zipfile = SharpCompress.Archive.ArchiveFactory.Open(fileName) let zipType = ProvidedTypeDefinition(asm, ns, typeName, Some(typeof\u0026lt;obj\u0026gt;)) ... for entry in zipfile.Entries do //we need to add types for each directory before adding the zipEntryType to the last occurrence let dirs = Path.getAllDirectories entry.FilePath let parent = processDirectories dirs zipType if entry.IsDirectory then parent.AddMembers \u0026lt;| mkProperties entry else let zipEntry = ProvidedTypeDefinition(safeTypeName entry.FilePath, Some(typeof\u0026lt;obj\u0026gt;)) zipEntry.AddMembers \u0026lt;| mkProperties entry parent.AddMember(zipEntry) zipType There are a few parts of code missing, but I\u0026rsquo;ll get to those in a second. You can see we create a root type to hold the type system that will represent the zip file:\nlet zipType = ProvidedTypeDefinition(asm, ns, typeName, Some(typeof\u0026lt;obj\u0026gt;)) SharpCompress is then used to open the archive and loop over the entries. For each file entry found we create a ProvidedTypeDefinition and corresponding properties and add it to the parent, but for each directory we only add properties to an existing ProvidedType.\nThe important functions missing here are mkProperties, getAllDirectories and processDirectories\nmkProperties # mkProperties is the meat and potatoes here:\nlet mkProperties (entry:IArchiveEntry) = [yield PP.MkStatic (\u0026#34;FilePath\u0026#34;, fun _ -\u0026gt; Expr.Value entry.FilePath) if not entry.IsDirectory then yield PP.MkStatic (\u0026#34;Crc\u0026#34;, fun _ -\u0026gt; Expr.Value entry.Crc) yield PP.MkStatic (\u0026#34;PackedSize\u0026#34;, fun _ -\u0026gt; Expr.Value entry.CompressedSize) yield PP.MkStatic (\u0026#34;Size\u0026#34;, fun _ -\u0026gt; Expr.Value entry.Size) yield PP.MkStatic (\u0026#34;CompressionRatio\u0026#34;, fun _ -\u0026gt; Expr.Value (float entry.Size / float entry.CompressedSize)) yield PP.MkStatic (\u0026#34;SpaceSavings\u0026#34;, fun _ -\u0026gt; Expr.Value (1.0 - float entry.CompressedSize / float entry.Size))] This function take a IArchiveEntry and returns a bunch of ProvidedProperties, one for each property we are interested in exposing in our type system. PP is a Type Abbreviation for ProvidedProperty, MkStatic is a Type Extension, these will both explained further on. In this function we are creating a list comprehension with each of the properties we want to represent. MkStatic is just a wrapper around the ProvidedProperty constructor, each property has a name, type and the getter function as represented by an expression. In this instance our expression is just the value of the property in IArchiveEntry so we represent this with Expr.Value entry.x. You might of been tempted to write this expression as \u0026lt;@@ entry.x @@\u0026gt; which uses the Untyped Quotation syntax but this would of resulted in an error from the compiler when in use. This is to do with type erasure, and the fact that only simple types can be represented as values in the quotation blocks. There\u0026rsquo;s a stackoverflow question that covers this too. The last two properties and not simple properties but calculations, that\u0026rsquo;s one of the beauties of Type Providers, you can easily leverage an existing API and make it more usable for your domain.\ngetAllDirectories # getAllDirectories is a module that extends Path so that a list of directory elements are returned for a path string. e.g. \u0026ldquo;/Users/dave/test\u0026rdquo; would yield [\u0026ldquo;Users\u0026rdquo;; \u0026ldquo;dave\u0026rdquo;; \u0026ldquo;test\u0026rdquo;]. We use this in processDirectories to ensure that each part of the path has a corresponding type stemming from the root. This ensures the ZipProvider provides the same hierarchy as a file browser. To be fair I\u0026rsquo;ve reinvented the wheel as this functionality is in Uri.Segments, but this serves as a how-to on extending existing type to bend them to your will! (That\u0026rsquo;s my excuse anyway!)\nmodule Path = let getAllDirectories (path:string) = let dname = Path.GetDirectoryName path dname.Split ([|Path.DirectorySeparatorChar|], StringSplitOptions.RemoveEmptyEntries) |\u0026gt; List.ofArray processDirectories # processDirectories is a recursive function that takes a list of directories and an initial base type and ensures that each directory has been assigned a type and a valid parent. Once the function has processed the entire path the last ProvidedTypeDefintion is returned from the function. You can see this used in buildTypes to either add files or directory properties as explained above. Sometime recursive functions can take a while to click in you brain, the secret here is in the acc or accumulator parameter which is the current parent that\u0026rsquo;s used to add the next type to.\nlet directoriesAdded = Dictionary\u0026lt;_,_\u0026gt; () let processDirectories directories (root:ProvidedTypeDefinition) = let rec loop list (acc:ProvidedTypeDefinition) = match list with | currentDir :: t -\u0026gt; if directoriesAdded.ContainsKey currentDir then loop t directoriesAdded.[currentDir] else //create provided type definition let pt = ProvidedTypeDefinition(currentDir, Some(typeof\u0026lt;obj\u0026gt;)) //add to parent provided type acc.AddMember pt //add to dictionary directoriesAdded.Add (currentDir, pt) //recurse loop t pt | [] -\u0026gt; (*return acc on completion*) acc loop directories root There\u0026rsquo;s also safeTypeName shown below, essentially this makes sure the type name is just the last segment of the path and that it doesn\u0026rsquo;t have leading or trailing slashes.\nlet safeTypeName name = //try get just the filename let filename = Path.GetFileName(name) //if it\u0026#39;s empty then it will be a directory if String.IsNullOrEmpty filename then name.Trim [|Path.DirectorySeparatorChar|] else filename And Another Thing \u0026hellip; # Oh I almost forgot, the type extension and type abbreviation I mentioned above, I used these to make things a little easier:\ntype PP = ProvidedProperty type ProvidedProperty = static member MkStatic\u0026lt;\u0026#39;a\u0026gt; (name, getter, ?setter) = let pp = PP (name, typeof\u0026lt;\u0026#39;a\u0026gt;, IsStatic = true, GetterCode = getter) setter |\u0026gt; Option.iter (fun v -\u0026gt; pp.SetterCode \u0026lt;- v) pp I added the MkStatic\u0026lt;'a\u0026gt; extension to slim down the code necessary to create a ProvidedProperty, without this the creation of a ProvidedProperty would be a little longer:\nlet pp = ProvidedProperty (\u0026#34;name\u0026#34;, typeof\u0026lt;mytype\u0026gt;, IsStatic = true, GetterCode = (fun _ -\u0026gt; Expr.Value 42)) It\u0026rsquo;s pure laziness, I get sick of typing typeof\u0026lt;'a\u0026gt; all the time, and the object initializer property names like GetterCode = .... The same goes for the type abbreviation. If I find myself typing a lot of repetitive long type names like ProvidedProperty then why not shorten it to PP. I do this when working with quotation types too.\nIf you wondering about the namespace I use, Xebec, its part of a suite of things I\u0026rsquo;ve been working on and off for a while involving lots of different things, it just my private codename I use\u0026hellip;\n99 Ways To Die # OK here\u0026rsquo;s all the code from top to bottom 99 lines. I don\u0026rsquo;t really like to duplicate but after the explanation about it will probably (hopefully) make sense now to read through.\nnamespace Xebex.Zip open System open System.Collections open System.Collections.Generic open System.IO open System.Reflection open System.Collections open Microsoft.FSharp open Microsoft.FSharp.Core.CompilerServices open Microsoft.FSharp.Quotations open ProviderImplementation.ProvidedTypes open SharpCompress.Archive module Path = let getAllDirectories (path:string) = let dname = Path.GetDirectoryName path dname.Split ([|Path.DirectorySeparatorChar|], StringSplitOptions.RemoveEmptyEntries) |\u0026gt; List.ofArray type PP = ProvidedProperty type ProvidedProperty = static member MkStatic\u0026lt;\u0026#39;a\u0026gt; (name, getter, ?setter) = let pp = PP (name, typeof\u0026lt;\u0026#39;a\u0026gt;, IsStatic = true, GetterCode = getter) setter |\u0026gt; Option.iter (fun v -\u0026gt; pp.SetterCode \u0026lt;- v) pp [\u0026lt;TypeProvider\u0026gt;] type public ZipProvider(cfg : TypeProviderConfig) as this = inherit TypeProviderForNamespaces() let asm = Assembly.GetExecutingAssembly() let ns = \u0026#34;Xebec\u0026#34; let root = ProvidedTypeDefinition(asm, ns, \u0026#34;ZipProvider\u0026#34;, Some(typeof\u0026lt;obj\u0026gt;)) let filePathParam = ProvidedStaticParameter(\u0026#34;FilePath\u0026#34;, typeof\u0026lt;string\u0026gt;) let buildTypes (typeName:string) (args:obj[]) = let fileName = args.[0] :?\u0026gt; string let zipfile = SharpCompress.Archive.ArchiveFactory.Open(fileName) let zipType = ProvidedTypeDefinition(asm, ns, typeName, Some(typeof\u0026lt;obj\u0026gt;)) let directoriesAdded = Dictionary\u0026lt;_,_\u0026gt; () let processDirectories directories (root:ProvidedTypeDefinition) = let rec loop list (acc:ProvidedTypeDefinition) = match list with | currentDir :: t -\u0026gt; if directoriesAdded.ContainsKey currentDir then loop t directoriesAdded.[currentDir] else //create provided type definition let pt = ProvidedTypeDefinition(currentDir, Some(typeof\u0026lt;obj\u0026gt;)) //add to parent provided type acc.AddMember pt //add to dictionary directoriesAdded.Add (currentDir, pt) //recurse loop t pt | [] -\u0026gt; (*return acc on completion*) acc loop directories root let safeTypeName name = //try get just the filename let filename = Path.GetFileName(name) //if it\u0026#39;s empty then it will be a directory if String.IsNullOrEmpty filename then name.Trim [|Path.DirectorySeparatorChar|] else filename let mkProperties (entry:IArchiveEntry) = [yield PP.MkStatic (\u0026#34;FilePath\u0026#34;, fun _ -\u0026gt; Expr.Value entry.FilePath) if not entry.IsDirectory then yield PP.MkStatic (\u0026#34;Crc\u0026#34;, fun _ -\u0026gt; Expr.Value entry.Crc) yield PP.MkStatic (\u0026#34;PackedSize\u0026#34;, fun _ -\u0026gt; Expr.Value entry.CompressedSize) yield PP.MkStatic (\u0026#34;Size\u0026#34;, fun _ -\u0026gt; Expr.Value entry.Size) yield PP.MkStatic (\u0026#34;CompressionRatio\u0026#34;, fun _ -\u0026gt; Expr.Value (float entry.Size / float entry.CompressedSize)) yield PP.MkStatic (\u0026#34;SpaceSavings\u0026#34;, fun _ -\u0026gt; Expr.Value (1.0 - float entry.CompressedSize / float entry.Size))] for entry in zipfile.Entries do //we need to add types for each directory before adding the zipEntryType to the last occurrence let dirs = Path.getAllDirectories entry.FilePath let parent = processDirectories dirs zipType if entry.IsDirectory then parent.AddMembers \u0026lt;| mkProperties entry else let zipEntry = ProvidedTypeDefinition(safeTypeName entry.FilePath, Some(typeof\u0026lt;obj\u0026gt;)) zipEntry.AddMembers \u0026lt;| mkProperties entry parent.AddMember(zipEntry) zipType do root.DefineStaticParameters ([filePathParam], buildTypes) do this.AddNamespace(ns, [root]) [\u0026lt;TypeProviderAssembly\u0026gt;] do() So if you made it this far you have seen: Type Providers, recursive functions, list comprehensions, type extensions, type abbreviations, object initialisers, pattern matching, and quotations, quite a few F# features!\nUntil next time!\n","date":"November 5, 2014","externalUrl":null,"permalink":"/programming/2014-11-05-flux-compression-redux/","section":"Blog","summary":"First of all the title, redux because I’m revising post I started on earlier in the year, compression because this has to do with compression, and Flux, which is also part of the redux, one of the first things I remember writing on the net was an article about Flux Compression Generators on H2G2, its still there too!\n","title":"Flux Compression (redux)","type":"programming"},{"content":"For any of you that are aware of the newly updated Xamarin Web site, you may have seen the following:\nObjective-C was ahead of its time 30 years ago. C# is ahead of its time today. Anything you can do in Objective-C or Java, you can do in C# with Xamarin—usually more succinctly and with fewer bugs.\nWhat is also true is that F# is way ahead of its time, and you can produce even more succinct code with even fewer bugs than C#!\nTake the code snippets from that page.\nFirst up the Objective C version:\n@interface Person : NSObject @property (strong, nonatomic) NSString *name; @end @implementation Person - (id)initWithName:(NSString *)name { self = [super init]; if (self) { self.name = name; } return self; } + (NSArray *)getNames { NSArray *people = @[ [[Person alloc] initWithName:@\u0026#34;David\u0026#34;], [[Person alloc] initWithName:@\u0026#34;Vinicius\u0026#34;], [[Person alloc] initWithName:@\u0026#34;Serena\u0026#34;], ]; NSMutableArray *names = [NSMutableArray array]; for (Person *person in people) { [names addObject:person.name]; } return names; } @end Heres the C# version:\nclass Person : NSObject { public string Name { get; set; } public static string[] GetNames() { var people = new[] { new Person { Name=\u0026#34;David\u0026#34; }, new Person { Name=\u0026#34;Vinicius\u0026#34; }, new Person { Name=\u0026#34;Serena\u0026#34; }, }; return people.Select(person =\u0026gt; person.Name).ToArray(); } } And finally the F# version:\ntype Person() = inherit NSObject() member val Name = \u0026#34;\u0026#34; with get, set static member GetNames() = [| new Person(Name=\u0026#34;David\u0026#34;) new Person(Name=\u0026#34;Vinicius\u0026#34;) new Person(Name=\u0026#34;Serena\u0026#34;) |] |\u0026gt; Array.map(fun person -\u0026gt; person.Name) You can see the F# version is doing exactly the same, although we are using the map function from the Array module rather than the Linq Select extension method.\nIts not all about the lines of code though, using F# gives you all many advantages:\nUsing the type system to make sure the code is behaving how you expect before you even compile. Pattern matching in F# is amazing! It can vastly simplify complex control logic, add Active patterns to that and you are ready to take on the world! Problems are approached from a functional perspective which often leads to succinct functions that are easy to reason about, test, and compose. F# emphasizes immutability and functional composition rather than inheritance, again this boils down to simplicity. Features like Type providers can vastly simplify how you deal with data within your application, making access to data really easy and intuitive. Theres are many areas that F# can really help productivity during development. I hope to write a few more short posts to really bring attention to these. I use F# all the time and often forget how awesome it is until I go back to another language thats missing those features.\nUntil next time!\n","date":"June 1, 2014","externalUrl":null,"permalink":"/programming/2014-06-01-anything-you-can-do/","section":"Blog","summary":"For any of you that are aware of the newly updated Xamarin Web site, you may have seen the following:\nObjective-C was ahead of its time 30 years ago. C# is ahead of its time today. Anything you can do in Objective-C or Java, you can do in C# with Xamarin—usually more succinctly and with fewer bugs.\n","title":"Anything you can do ...","type":"programming"},{"content":"","date":"June 1, 2014","externalUrl":null,"permalink":"/tags/csharp/","section":"Tags","summary":"","title":"Csharp","type":"tags"},{"content":"With the release of Xamarin 3 there is a swathe of new features to the platform, but obviously the most important one is obviously F# support is now included by default in Xamarin Studio so there is no escape from the awesomeness of F#!\nSo what else have we got, well loads of other goodies too like Xamarin Designer for iOS, Xamarin.Forms, Major IDE enhancements, Improved code sharing with PCL and Shared projects, and BCL Documentation. You can see the release blog post here: announcing Xamarin 3\nThere are tons of things I could show you, but for this post lets have a quick look at Xamarin.Forms\nXamarin.Forms # So what is Xamarin.Forms? Well to paraphrase the Xamarin blurb a little bit:\nBuild native UIs for iOS, Android and Windows Phone from a single, shared codebase. Xamarin.Forms pages represent single screens within an app.\nPages contain layouts, buttons, labels, lists, and other common controls. Connect these controls to shared backend code and you get fully native iOS, Android, and Windows Phone apps built entirely with shared code.\nSo as this just a whistle-stop post I\u0026rsquo;ll quickly show you the code.:\nlet profilePage = ContentPage(Title=\u0026#34;Profile\u0026#34;, Icon=\u0026#34;Profile.png\u0026#34;, Content=StackLayout.Create( [ Entry (Placeholder=\u0026#34;Username\u0026#34;) Entry (Placeholder=\u0026#34;Password\u0026#34;, IsPassword=true) Button (Text=\u0026#34;Login\u0026#34;, TextColor=Color.White, BackgroundColor=Color.FromHex \u0026#34;77D065\u0026#34;) ], spacing=20.0, padding=thickness 50.0, verticalOptions=LayoutOptions.Center)) let settingsPage = ContentPage(Title=\u0026#34;Settings\u0026#34;, Icon=\u0026#34;Settings.png\u0026#34;) let mainPage = TabbedPage.create [profilePage;settingsPage] The result is a nice cross platform UI thats easy to build and maintain, sweet!\nYou can see by the code thats there are several components at play but it\u0026rsquo;s very easy to see how the code relates to the UI layout. A stack layout with two Entry types followed by a Button. You can also see at the bottom that there is a TabbedPage comprising of the profilePage and the settingPage.\nNice and easy!\nOK, thats it for now, but next time I\u0026rsquo;ll be delving in a little deeper and showing you exactly how to build a PCL backed F# app over multiple platforms.\nUntil next time!\n","date":"May 28, 2014","externalUrl":null,"permalink":"/programming/2014-05-28-xamarin-3-fsharp-awsomeness/","section":"Blog","summary":"With the release of Xamarin 3 there is a swathe of new features to the platform, but obviously the most important one is obviously F# support is now included by default in Xamarin Studio so there is no escape from the awesomeness of F#!\n","title":"Xamarin 3 F# Awesomeness","type":"programming"},{"content":" Over the last year a lot of work has been done on the F# addin for Xamarin Studio. Lots of great new features have been added and a lot of bugs have been squashed. I want to talk a bit about whats been happening and the evolution of the F# addin.\nThe F# addin for MonoDevelop / Xamarin Studio hasn\u0026rsquo;t always been as stable and pretty as it is now. When I first started working on it I went through a rocky phase of just trying to get it to compile and install. There were a whole host of changes going on in MonoDevelop which caused a lot of head scratching and late nights. Of all the issues encountered tooltips were the absolute bane of my life, they were often horribly mangled like this:\nOr completion lists were insanely long and downright pugly:\nModern IDE\u0026rsquo;s have features like tooltips and auto completions which are so intrinsic that without them you feel a little lost without them, instead you have to rely on having eidetic memory of the various APIs and functions. Over the course of fixing bugs and adding new features inevitably you end up breaking lots of things which can make for a frustrating experience with half mangled IDE. Thankfully all these issues have now been addressed and lots of new features have also been added. Here are all the new features and improvements that have gone in over the last year or so.\nUI Improvements # Rename refactoring # One of the most recent and exciting new features is rename refactoring. Renaming can be achieved by either using a shortcut key Meta R or by right clicking and selecting rename from the context menu. Here\u0026rsquo;s a screen cast of rename refactoring in action:\nHighlight usages # The precursor to rename refactoring was highlight usages, by clicking in the code editor the current identifier is highlighted and all occurrences of of that identifier are also highlighted. This makes it very easy to see where a specific identifier is used and is one of my favorite features:\nTooltip overload refinements # Overload tooltips have now been refined so they look like the image below rather than the huge overload list shown at the top of this post.\nTooltip arrow indicators # A little arrow indicators is now shown and positioned over the center of the tooltip. Yeah is only cosmetic but it was annoying having it missing.\nSymbol tooltips # We now also have tooltips for symbols, which makes custom symbol use more palatable. Now you can simply hover over the symbol see see its signature and xmldoc summary.\nDrag and drop file ordering # F# source files can now be dragged in the Solution pad for correct ordering. This is limited to ordering files within a single folder at the moment. Source files can still be moved from one folder to another but they must be ordered as a separate operation.\nPathed document navigation # Pathed document navigation is implemented in the source window to aid navigation between modules, types and functions within a source file:\nInteractive Improvements # Send all project references to FSI # From the context menu or keyboard shortcut Ctrl Meta P you can choose to send all of the current projects reference to F# interactive, this saves having to enter a bunch of #r statements.\nSend line moves down automatically # For doing demos and testing scripts you can use a keyboard shortcut to sent the current line to F# interactive, the caret automatically drops down to the next line which makes stepping through the script a pinch.\nClear/Reset FSI keyboard shortcuts # By pressing Ctrl Meta R you can reset the current instance of F# interactive, similarly Ctrl Meta C will clear the current F# interactive window.\nFSI theming # You can now also choose to have F# interactive match you current theme colours or pick your own:\nEngineering improvements # Originally we had a reflective binding to the F# compiler services which meant it was version agnostic but quite difficult to debug and add new features to.\nActually just for posterity, and you might find this useful/interesting heres the code, it uses the dynamic lookup operator (?) :\nlet (?) (o:obj) name : \u0026#39;R = // The return type is a function, which means that we want to invoke a method if FSharpType.IsFunction(typeof\u0026lt;\u0026#39;R\u0026gt;) then let argType, resType = FSharpType.GetFunctionElements(typeof\u0026lt;\u0026#39;R\u0026gt;) FSharpValue.MakeFunction(typeof\u0026lt;\u0026#39;R\u0026gt;, fun args -\u0026gt; // We treat elements of a tuple passed as argument as a list of arguments // When the \u0026#39;o\u0026#39; object is \u0026#39;System.Type\u0026#39;, we call static methods let methods, instance, args, owner = let args = if Object.Equals(argType, typeof\u0026lt;unit\u0026gt;) then [| |] elif not(FSharpType.IsTuple(argType)) then [| args |] else FSharpValue.GetTupleFields(args) if (typeof\u0026lt;System.Type\u0026gt;).IsAssignableFrom(o.GetType()) then let methods = (unbox\u0026lt;Type\u0026gt; o).GetMethods(staticFlags) |\u0026gt; Array.map asMethodBase let ctors = (unbox\u0026lt;Type\u0026gt; o).GetConstructors(ctorFlags) |\u0026gt; Array.map asMethodBase let owner = (unbox\u0026lt;Type\u0026gt; o).Name + \u0026#34; (static)\u0026#34; Array.concat [ methods; ctors ], null, args, owner else let owner = o.GetType().Name + \u0026#34; (instance)\u0026#34; o.GetType().GetMethods(instanceFlags) |\u0026gt; Array.map asMethodBase, o, args, owner // A simple overload resolution based on the name and number of parameters only let methods = [ for m in methods do if m.Name = name \u0026amp;\u0026amp; m.GetParameters().Length = args.Length then yield m if m.Name = name \u0026amp;\u0026amp; m.IsGenericMethod \u0026amp;\u0026amp; m.GetGenericArguments().Length + m.GetParameters().Length = args.Length then yield m ] match methods with | [] -\u0026gt; failwithf \u0026#34;No method \u0026#39;%s\u0026#39; with %d arguments found in %s\u0026#34; name args.Length owner | _::_::_ -\u0026gt; failwithf \u0026#34;Multiple methods \u0026#39;%s\u0026#39; with %d arguments found %s\u0026#34; name args.Length owner | [:? ConstructorInfo as c] -\u0026gt; c.Invoke(args) | [ m ] when m.IsGenericMethod -\u0026gt; let tyCount = m.GetGenericArguments().Length let tyArgs = args |\u0026gt; Seq.take tyCount let actualArgs = args |\u0026gt; Seq.skip tyCount let gm = (m :?\u0026gt; MethodInfo).MakeGenericMethod [| for a in tyArgs -\u0026gt; unbox a |] gm.Invoke(instance, Array.ofSeq actualArgs) | [ m ] -\u0026gt; m.Invoke(instance, args) ) |\u0026gt; unbox\u0026lt;\u0026#39;R\u0026gt; else // When the \u0026#39;o\u0026#39; object is \u0026#39;System.Type\u0026#39;, we access static properties let typ, flags, instance = if (typeof\u0026lt;System.Type\u0026gt;).IsAssignableFrom(o.GetType()) then unbox o, staticFlags, null else o.GetType(), instanceFlags, o // Find a property that we can call and get the value let prop = typ.GetProperty(name, flags) if Object.Equals(prop, null) then // Find a field that we can read let fld = typ.GetField(name, flags) if Object.Equals(fld, null) then // Try nested type... let nested = typ.Assembly.GetType(typ.FullName + \u0026#34;+\u0026#34; + name) if Object.Equals(nested, null) then failwithf \u0026#34;Property, field or nested type \u0026#39;%s\u0026#39; not found in \u0026#39;%s\u0026#39; using flags \u0026#39;%A\u0026#39;.\u0026#34; name typ.Name flags elif not ((typeof\u0026lt;\u0026#39;R\u0026gt;).IsAssignableFrom(typeof\u0026lt;System.Type\u0026gt;)) then failwithf \u0026#34;Cannot return nested type \u0026#39;%s\u0026#39; as value of type \u0026#39;%s\u0026#39;.\u0026#34; nested.Name (typeof\u0026lt;\u0026#39;R\u0026gt;.Name) else nested |\u0026gt; box |\u0026gt; unbox\u0026lt;\u0026#39;R\u0026gt; else // Get field value fld.GetValue(instance) |\u0026gt; unbox\u0026lt;\u0026#39;R\u0026gt; else // Call property let meth = prop.GetGetMethod(true) if prop = null then failwithf \u0026#34;Property \u0026#39;%s\u0026#39; found, but doesn\u0026#39;t have \u0026#39;get\u0026#39; method.\u0026#34; name try meth.Invoke(instance, [| |]) |\u0026gt; unbox\u0026lt;\u0026#39;R\u0026gt; with err -\u0026gt; failwithf \u0026#34;Failed to get value of \u0026#39;%s\u0026#39; property (of type \u0026#39;%s\u0026#39;), error: %s\u0026#34; name typ.Name (err.ToString()) To use it a type wrapper had to be constructed a bit like this:\nlet interactiveCheckerType = asmCompiler.GetType(\u0026#34;Microsoft.FSharp.Compiler.SourceCodeServices.InteractiveChecker\u0026#34;) type InteractiveChecker(wrapped:obj) = member x.TryGetRecentTypeCheckResultsForFile(filename:string, options:CheckOptions) = let res = wrapped?TryGetRecentTypeCheckResultsForFile(filename, options.Wrapped) : obj if res = null then None else let tuple = res?Value Some(UntypedParseInfo(tuple?Item1), TypeCheckResults(tuple?Item2), int tuple?Item3) It was quite fiddly to get right as FSharp.Core had to be used reflectively as you would potentially be using a different version of FSharp.Core depending on what version of F# you were binding to. I think this kind of reflective calling could be simplified by creating an F# type provider, I can think of a variety of uses for something like this.\nUltimately a branch called hardbinding was created which led to the creation of the F# Compiler Editor branch of the F# compiler. Slowly over the last few months that evolved into the FSharp Compiler Service aka FCS.\nI also experimented quite a bit with a custom tokeniser that allowed the keywords and other syntax elements to be highlighted as tooltips:\nAt the time the F# binding was going through quite a turbulent change to the source code, and the combination of trying to develop FSharp.Compiler.Services and also refactoring the existing code meant that I decided to put this on hold for the time being. I hope to resurrected this work one day, I can see that keyword highlighting would be very useful to beginners, especially if a detailed description was available similar to what is available on MSDN. It also addresses some bugs that are still outstanding so it would be nice to get finished.\nOver the last couple of months a lot of the common code around parsing of long identifiers has been refactored so that is can be shared with other tools. In the F# Binding we also have an emacs plugin which uses this shared code. We have also created a shared language agent which makes using these language services even easier still!\nSwat all the bugs! # Anyone remember Picnic paranoia? Swat all the bugs A shed load of bugs have also been swatted. Using the F# addin is no longer like ignoring the Unstable Structure sign and hoping things don\u0026rsquo;t crumble under your feet. Admittedly there is still work to be done but some of the recent refactoring make it even easier to contribute to either new features or fixing bugs. We\u0026rsquo;ve also added the F# binding to up-for-grabs.net and added the up-for-grabs issues as a starting point for anyone thinking about to helping out. There\u0026rsquo;s been a lot of blood sweat and tears and its consumed quite a lot of my spare time but I hope you all enjoy using the new F# addin features.\nLastly a big thank you to everyone who has helped contribute over the last year, keep the contributions coming in!\nThat\u0026rsquo;s all for now, see you next time!\n","date":"February 10, 2014","externalUrl":null,"permalink":"/programming/2014-02-10-danger-unstable-structure/","section":"Blog","summary":" Over the last year a lot of work has been done on the F# addin for Xamarin Studio. Lots of great new features have been added and a lot of bugs have been squashed. I want to talk a bit about whats been happening and the evolution of the F# addin.\n","title":"Danger unstable structure - No more!","type":"programming"},{"content":"Only a quick post this week. Last time we looked at SpriteKit and how to add some particle emmiters to simulate a star-field and exhaust on a spaceship, this time lets look at adding some touch based input to move the spaceship around. The first thing we need to do is add a type of gesture recogniser, there are various built in gestures:\nUITapGestureRecognizer UIPinchGestureRecognizer UIRotationGestureRecognizer UISwipeGestureRecognizer UIPanGestureRecognizer UIScreenEdgePanGestureRecognizer UILongPressGestureRecognizer From the names above it\u0026rsquo;s pretty easy to get a feel for how they should be used, you can create your own subclass of UIGestureRecognizer if you need a custom one.\nGesture recognizers come in two types continuous and discrete. A discrete gesture is single action like tap or double tap and results in a single action been sent. A continuous gesture is like pan, swipe, or rotate which is interpreted as a series of messages being sent.\nFor our purposes we are going to be using the UIPanGestureRecognizer which is a continuous gesture. What we need to do is create a function that sets up the UIPanGestureRecognizer ready for us to use. We do that by creating an instance of the UIPanGestureRecognizer and add it to our view:\nlet setupGestures() = use panRecogniser = new UIPanGestureRecognizer(x, MonoTouch.ObjCRuntime.Selector(\u0026#34;PanSelector\u0026#34;)) x.View.AddGestureRecognizer(panRecogniser) Here we are also using a selector, which means we can use an attribute like [\u0026lt;Export(\u0026quot;PanSelector\u0026quot;)\u0026gt;] to define the function that will be used as the callback. Lets define that function now:\nlet OnLabelPan( sender: UIGestureRecognizer) = match sender with | :? UIPanGestureRecognizer as pan -\u0026gt; match pan.State with | UIGestureRecognizerState.Changed -\u0026gt; let movement = pan.TranslationInView(x.View) let move = SKAction.MoveBy(movement.X * 1.75f, -movement.Y * 1.75f, 0.05) let ship = scene.GetChildNode(\u0026#34;Ship\u0026#34;) ship.RunAction(move) pan.SetTranslation(PointF.Empty, x.View) | _ -\u0026gt; () | _ -\u0026gt; () First of all we use pattern matching to do a type match | :? UIPanGestureRecognizer as pan -\u0026gt;. This ensures we are dealing with the UIPanGestureRecognizer type. We might of applied multiple gesture recognisers to the view like swipe and rotate and had this function deal with all of them, we can handle this nicely with the type match.\nWe can now use a pattern match on the state of the gesture recogniser to react to just the changed event UIGestureRecognizerState.Changed.\nAs mentioned previously the pan gesture is continuous and will send a changed action whenever the finger moves on the screen, this gives us a chance to retrieve the current translation of the pan in the current view. We do this by calling pan.TranslationInView(x.View). We can now apply a movement to our spaceship sprite by creating an action using SKAction.MoveBy. We multiply the translation retrieved by 1.75 to allow for the initial distance the the pan gesture moves before triggering. We also invert the Y axis so that the spaceship sprite moves in the correct Y direction. The final parameter is the time the action runs for, we use a really small time of 0.05 (50ms). This stops the spaceship sprite from moving like an ordinary mouse pointer, just enough inertia to make it feel smooth.\nTo apply the action to the spaceship all we need to do is retrieve it from the scene using scene.GetChildNode and call the RunAction function passing in the action we just created.\nFinally we set the pan translation back to zero using: pan.SetTranslation(PointF.Empty, x.View), this ensures that the spaceship only moves by last changed translation action. Failure to reset the translation would result in the spaceship having too much inertia from the previous actions making it very difficult to control.\nWe could also use another overload of UIPanGestureRecognizer which takes an Action\u0026lt;UIPanGestureRecognizer\u0026gt;, we can pass this in as a lambda function:\nlet setupGestures() = use panRecogniser = new UIPanGestureRecognizer (fun (pan:UIPanGestureRecognizer) -\u0026gt; match pan.State with | UIGestureRecognizerState.Changed -\u0026gt; let movement = pan.TranslationInView(x.View) let move = SKAction.MoveBy(movement.X * 1.75f, -movement.Y * 1.75f, 0.05) let ship = scene.GetChildNode(\u0026#34;Ship\u0026#34;) ship.RunAction(move) pan.SetTranslation(PointF.Empty, x.View) | _ -\u0026gt; ()) I think the attribute based version is a little cleaner as you can move the callback functionality away from the definition. To be honest I don\u0026rsquo;t mind either way, although if the lambda definition gets too big you will definitely be better off with the former.\nFinally we need to plug in the setupGestures function, we do that by calling it at the end of ViewDidLoad:\noverride x.ViewDidLoad () = base.ViewDidLoad() setupScene() setupGestures() Here\u0026rsquo;s a quick YouTube video so you can see this in action:\nIf you want to check out the project then you can find it in my GitHub repo .\nThat\u0026rsquo;s all for now, see you next time!\n","date":"September 29, 2013","externalUrl":null,"permalink":"/programming/2013-09-29-adding-touch-to-spritekit/","section":"Blog","summary":"Only a quick post this week. Last time we looked at SpriteKit and how to add some particle emmiters to simulate a star-field and exhaust on a spaceship, this time lets look at adding some touch based input to move the spaceship around. ","title":"Adding Touch To SpriteKit","type":"programming"},{"content":"","date":"September 29, 2013","externalUrl":null,"permalink":"/tags/games/","section":"Tags","summary":"","title":"Games","type":"tags"},{"content":"I have been meaning to write this post for quite a while now. Since the first announcement of the iOS7 beta I immediately saw the list of new API\u0026rsquo;s and SpriteKit caught my eye straight away. I only managed to get time to briefly look over the API and saw that is wasn\u0026rsquo;t the usual trashy API with a million method overloads, internal mutation sucker punch type thing. It seems to be very declarative and intuitive, which makes for a nice change. First of all lots of kudos to Xamarin for getting Xamarin iOS 7 out so swiftly, you can read about some of the new features here.\nSpriteKit # So what is SpriteKit?\nSpriteKit, as you might have guessed, is a games oriented API aimed at getting you quickly up and running with 2D sprites so that you can spend more time building your games rather than mucking about with the low level stuff. Lets have a really quick tour of what\u0026rsquo;s in there, I don\u0026rsquo;t want to spend long on this as you can read the programming guide on the Apple site for further details.\nScenes # SpriteKit represents the different parts of your games with a scenes. A scene could be the title screen or the levels in the game. A scene is really just a just a collection of nodes which represents all of the objects currently visible. There are several different types of node that can be used in the scene.\nNodes # Several different type of node are available to use, here they are:\nSKVideoNode - Allows videos to be embedded into the scene. SKCropNode - A crop node allows you to mask of different areas of the viewing area. SKEffectNode - The effects node allows its children to be rendered into a private frame buffer where Core Image effects can be applied before being blended back into the main scene. SKEmitterNode - Allows particles to be placed into a scene. SKLabelNode - Allows for arbitrary text to be places into the scene. SKShapeNode - Allows path based shapes to be draw in the scene. SKSpriteNode - This is your standard textured image which can be colour blended, scaled, rotated etc. SKNode - This is the base node type from which all the others are derived. Transitions # Transitions allow you to move from one scene to another, allowing for a animated effect to be applied during the transition.\nActions # Actions allow you to declarative apply an action to a node. For example you could write something like this:\nSKAction *moveUp = [SKAction moveByX:0 y:100.0 duration:1.0]; SKAction *zoom = [SKAction scaleTo:2.0 duration:0.25]; SKAction *wait = [SKAction waitForDuration: 0.5]; SKAction *fadeAway = [SKAction fadeOutWithDuration:0.25]; SKAction *removeNode = [SKAction removeFromParent]; SKAction *sequence = [SKAction sequence:@[moveUp, zoom, wait, fadeAway, removeNode]]; [node runAction: sequence]; This creates a sequence of actions: moveUp, zoom, wait, fadeAway, removeNode. The actions are reusable and stateless so they can be applied to any nodes in the scene. If we didn\u0026rsquo;t want the actions to be applied in a sequence we could use a group, which applies the actions in parallel.\nDon\u0026rsquo;t worry if I\u0026rsquo;ve scared you with Objective-C there will be none of that when we get into writing a little demo in a moment.\nPhysics # The physics part of SpriteKit can be really fun to play with, its fairly easy to fill a screen full of cubes and bash them about watching the gravity and collision effects. The physics engine looks like its based on Box2D and involves adding approximate shapes for your game objects and then adding a bunch of physical properties like mass, friction, linear damping, restitution etc.\nFirst Steps # That was a really quick whistle stop tour just to give you a flavour of what\u0026rsquo;s in there. For this post we are going to look at the SKEmitterNode and see what we can do.\nThe first thing to do is set up the skeleton, not an actual skeleton mind, just the skeleton of the demo.\nCreate a new F# iOS Single View Application. SpriteKit uses a subclass of UIView for its rendering surface and is controlled as usual by the UIViewController but we need to add a few things and make a few chnages to start using SpriteKit:\nWe need to open references for MonoTouch.SpriteKit and MonoTouch.CoreGraphics. Add a couple of virtual overloads for ViewDidAppear and ViewDidDisappear. Add an instance of SKScene and SKView. We will also add a function called setupScene which will initialise the scene, this will be called from DidViewLoad. That means something like this will do the trick for an empty scene:\nnamespace SpriteKitSingleView open System open System.Drawing open MonoTouch.Foundation open MonoTouch.UIKit open MonoTouch.SpriteKit open MonoTouch.CoreGraphics [\u0026lt;Register (\u0026#34;SpriteKitViewController\u0026#34;)\u0026gt;] type SpriteKitViewController () as x= inherit UIViewController () let mutable scene = Unchecked.defaultof\u0026lt;SKScene\u0026gt; let mutable spriteView = new SKView() let setupScene() = spriteView.Bounds \u0026lt;- RectangleF(0.f, 0.f, x.View.Bounds.Width * UIScreen.MainScreen.Scale, x.View.Bounds.Height * UIScreen.MainScreen.Scale) spriteView.ShowsDrawCount \u0026lt;- true spriteView.ShowsNodeCount \u0026lt;- true spriteView.ShowsFPS \u0026lt;- true x.View \u0026lt;- spriteView scene \u0026lt;- new SKScene (spriteView.Bounds.Size, BackgroundColor = UIColor.Blue, ScaleMode = SKSceneScaleMode.AspectFit) override x.DidReceiveMemoryWarning () = base.DidReceiveMemoryWarning () override x.ShouldAutorotateToInterfaceOrientation (orientation) = orientation \u0026lt;\u0026gt; UIInterfaceOrientation.PortraitUpsideDown override x.ViewDidLoad () = base.ViewDidLoad() setupScene() override x.ViewDidAppear(animated) = base.ViewDidDisappear (animated) spriteView.PresentScene(scene) override x.ViewDidDisappear(animated) = base.ViewDidDisappear (animated) scene.RemoveAllChildren() scene.RemoveAllActions() The main interesting bit here is setupScene. The spriteView instance is created at the beginning of the SpriteKitViewController's implicit constructor. We need to defer creating the scene until later because in the constructor the UIView has not yet been initialised by the the framework and we would get a null reference exception.\nThe first thing we do is get the dimensions of the current view, multiply it by the current scale, then apply it to the spriteView Bounds property. The scale property is used for the various DPI modes in iOS devices. Next we add a few debug outputs to the spriteView to show the current draw and node counts as well at the current frame rate. We assign the spriteView to the View property of the ViewController. And finally we create a new SKScene, assigning the view boundary, background colour and the scaling mode.\nAs a side note we could create a storyboard and use an SKView as the custom class instead of the default UIView, doing this way means that when the ViewDidLoad overload is called the View property of the SpriteKitViewControl would already be initialized with a SKView. This is a little more tricky in F# as we don\u0026rsquo;t currently have the fancy UI designer integration in Xamarin Studio. You would have to do this in Xcode and copy it to your project manually.\nAdding the Sprite # Next we will add a spaceship sprite:\nWe need to add a .png file to the project. Create an instance of an SKSprite. We also need to add the sprite to the scene. Add a spaceship texture to the project and make sure that the build action is set to BundleResource\nNext add the following to code to the setupScene function:\nuse sprite = new SKSpriteNode (\u0026#34;Art/viper_mark_vii.png\u0026#34;) sprite.Position \u0026lt;- PointF (scene.Frame.GetMidX(), scene.Frame.GetMidY()) sprite.Name \u0026lt;- \u0026#34;Ship\u0026#34; scene.AddChild(sprite) I added my sprite to a sub folder in the project called Art, once you start adding lots of graphics assets you probably want to make sure they are properly organised. Next I set the spaceship\u0026rsquo;s initial position to the center of the current scene using the scene.Frame.GetMidX() and scene.Frame.GetMidY() methods. We give the sprite node a name using the Name property. This is useful if we want to refer to the spaceship via its name in the node graph rather than using its object reference Finally we add the sprite to the scene using scene.AddChild(sprite).\nSo this now gives us a single spaceship sitting in the middle of the screen:\nCreating Particles with xCode\u0026rsquo;s Particle Designer # The great thing about SpriteKit is it also comes with a nice particle designer. To use the particle designer you have to fire up Xcode and add a new file of type SpriteKit Particle Designer. You get a choice of a eight different preset\u0026rsquo;s particle types: Bokeh, Fire, Fireflies, Magic, Rain, Smoke, Snow, and Spark.\nIt certainly saves a lot of time developing the particle effects with the particle designer, there are loads of parameters to play around with. If I quickly choose the rain preset and fiddle with the parameters a bit you get a star-field type effect like this:\nWhile we\u0026rsquo;re here lets also create an exhaust plume for our spaceship, create another particle, this time using the spark preset and tweak it so it look a bit like this:\nAdding The Particles # We now have all we need to plug the particles into our demo. Find the particles you created in Xcode and copy or move them into your project, don\u0026rsquo;t forget to make sure that the build action to BundleResource.\nWhile I remember lets change the background colour to black, the blue looks a bit lurid and the exhaust trail wont look its best against a blue background. Find where scene is initialised in setupScene and change it so it looks like this:\nscene \u0026lt;- new SKScene (spriteView.Bounds.Size, BackgroundColor = UIColor.Black, ScaleMode = SKSceneScaleMode.AspectFit) If you look at the SKEmitterNode constructor you might be slightly befuddled by the fact that it only takes either an NSCoder, NSObjectFlag or a nativeint. To help us out we create a nice little function to do the dirty work for us:\nmodule spritekit = type SKEmitterNode with static member fromResource res = let emitterpath = NSBundle.MainBundle.PathForResource (res, \u0026#34;sks\u0026#34;) NSKeyedUnarchiver.UnarchiveFile(emitterpath) :?\u0026gt; SKEmitterNode The .sks files produced by Xcode are archive files so we need to get them into a format that works in our project. First we find the full path for the resource as its embedded in our app bundle - NSBundle.MainBundle.PathForResource (res, \u0026quot;sks\u0026quot;), next we use the UnarchiveFile method from the NSKeyedUnarchiver type to get an NSObject. We finally cast the NSObject as an SKEmitterNode before it is returned.\nThe function shown above is added as a static extension method to the SKEmitterNode. We could of also added this using a module or even just a simple function but at the moment we don\u0026rsquo;t have a clear view of any other extensions that we might need so we\u0026rsquo;ll just keep it tucked up in the SKEmitterNode type for now.\nWe can now use this function to load and add our star-field to our scene. Add this piece of code before the spaceship code we added previously:\nuse stars = SKEmitterNode.fromResource \u0026#34;Stars\u0026#34; stars.Position \u0026lt;- PointF(scene.Frame.GetMidX(), scene.Frame.GetMaxY()) scene.AddChild(stars) Its pretty simple now we have the helper function to load the star-field as a resource. Notice we add the star-field as a child of the scene and position it at the middle X coordinate of the screen (scene.Frame.GetMidX()) and the maximum Y coordinate (scene.Frame.GetMaxY()). This places the star-field centrally at the top of the screen.\nWe can now go ahead and add our exhaust plume in the same way:\nuse flame = SKEmitterNode.fromResource \u0026#34;Fire\u0026#34; flame.Position \u0026lt;- PointF(0.f, -60.f) sprite.AddChild(flame) The only difference here is we position the exhaust at location X = 0.0, Y = -60.0 and add the exhaust as a child of the spaceship. This means that the exhaust is offset -60.0 in the Y axis from the spaceships location, this is because child nodes inherit their parents coordinate system. This makes groups of sprites easy to animate and manipulate as you don\u0026rsquo;t have to work out all the offsets.\nIf we run our demo now it starts to look more interesting:\nThat\u0026rsquo;s all for now, I hope you have enjoyed this little look at particles in SpriteKit. If you want to download the demo project you can find it in my GitHub repo.\nUntil next time!\n","date":"September 20, 2013","externalUrl":null,"permalink":"/programming/2013-09-20-sprite-kit-particle-fun/","section":"Blog","summary":"I have been meaning to write this post for quite a while now. Since the first announcement of the iOS7 beta I immediately saw the list of new API’s and SpriteKit caught my eye straight away. I only managed to get time to briefly look over the API and saw that is wasn’t the usual trashy API with a million method overloads, internal mutation sucker punch type thing. It seems to be very declarative and intuitive, which makes for a nice change. ","title":"Spritekit particle fun","type":"programming"},{"content":" There\u0026rsquo;s been a fair bit of activity lately from a project called ScriptCS, it allows you to put together a project using C# as a lightweight scripting language, forgoing the use of Visual Studio which can sometimes be too bloated and bulky.\nIt also allows you to use C# in a Read Evaluate Print Loop - REPL. This is nothing new to F# and indeed lots of other languages have REPL\u0026rsquo;s too. One of the other benefits of ScriptCs is that it also integrates nicely with Nuget allowing you to use your favourite libraries quite easily. Finally there are Script Sacks which can be used to further reduce the amount of code you need to write when working with common frameworks.\nIt would be nice to leverage some of this new functionality from F#, and I don\u0026rsquo;t like to see F# left out, especially when F# already has a REPL environment and is a really great language for scripting.\nThe F# compiler is also open source so we can utilize the code to add various tooling and features like refactoring, formatting, and analysis. See Fantomas, FSharp-Refactor, and the FSharpBinding for more details.\nLets get to work # The interface for adding a new script engines looks like this:\npublic interface IScriptEngine { string BaseDirectory { get; set; } ScriptResult Execute(string code, string[] scriptArgs, IEnumerable\u0026lt;string\u0026gt; references, IEnumerable\u0026lt;string\u0026gt; namespaces, ScriptPackSession scriptPackSession); } First lets create a new namespace and open up all the namespaces we need:\nnamespace ScriptCs.Engine.FSharp open ScriptCs open Common.Logging open System open System.IO open System.Collections.Generic open Microsoft.FSharp.Compiler.Interactive.Shell open ExtCore open System.Linq Next we need to store the result of an attempted expression evaluation. When we send code to an interactive session we might be in one of three different states:\nIncomplete - We have entereed a line but the expression is not complete, in F# we use ;; to indicate the end of an expression. Error - The entered expression resulted in an error. Success - The expression that was entered was evaluated successfully. We can model this with a discriminated union like this:\ntype Result = | Success of String | Error of string | Incomplete Next we will augment the existing FsiEvaluationSession type to allow it to be encapsulated and used by our interface.\ntype FSharpEngine(host:ScriptHost) = let stdin = new StreamReader(System.IO.Stream.Null) let stdoutStream = new CompilerOutputStream() let stdout = StreamWriter.Synchronized(new StreamWriter(stdoutStream, AutoFlush=true)) let stderrStream = new CompilerOutputStream() let stderr = StreamWriter.Synchronized(new StreamWriter(stderrStream, AutoFlush=true)) let getOutput (session: FsiEvaluationSession) code = let tryget() = let error = stderrStream.Read() if error.Length \u0026gt; 0 then Error(error) else Success(stdoutStream.Read()) try session.EvalInteraction(code) if code.EndsWith \u0026#34;;;\u0026#34; then tryget() else Incomplete with ex -\u0026gt; Error ex.Message let commonOptions = [| \u0026#34;fsi.exe\u0026#34;; \u0026#34;--nologo\u0026#34;; \u0026#34;--readline-\u0026#34;|] let session = FsiEvaluationSession(commonOptions, stdin, stdout, stderr) let (\u0026gt;\u0026gt;=) (d1:#IDisposable) (d2:#IDisposable) = {new IDisposable with member x.Dispose() = d1.Dispose(); d2.Dispose()} member x.Execute(code) = getOutput session code member x.AddReference(ref) = session.EvalInteraction(sprintf \u0026#34;#r @\\\u0026#34;%s\\\u0026#34;\u0026#34; ref) member x.SilentAddReference(ref) = x.AddReference(ref) stdoutStream.Read() |\u0026gt; ignore member x.ImportNamespace(namespace\u0026#39;) = session.EvalInteraction(sprintf \u0026#34;open %s\u0026#34; namespace\u0026#39;) member x.SilentImportNamespace(namespace\u0026#39;) = x.ImportNamespace(namespace\u0026#39;) stdoutStream.Read() |\u0026gt; ignore interface IDisposable with member x.Dispose() = (stdin \u0026gt;\u0026gt;= stdoutStream \u0026gt;\u0026gt;= stdout \u0026gt;\u0026gt;= stderrStream \u0026gt;\u0026gt;= stderr).Dispose() We are mainly just wrapping the FsiEvaluationSession here, adding convenience methods to evaluate code and gather the output from the compiler streams. The current open source implementation of FsiEvaluationSession uses streams to add input and receive the output and errors. Stream processing makes sense when you are just dealing with a Console with in, out, and error streams, but it gets decidedly more complex if you want deterministic evaluation. Stream observation, polling, and looking for termination characters is fairly awkward to get right.\nAfter a few conversations with Don Syme he kindly assisted with an experimental version of FsiEvaluationSession that allowed expressions to be evaluated using the EvalInteraction and EvalExpression functions rather than writing directly to the input stream. I\u0026rsquo;m very grateful for the work Don has done so far to help me with this. I think more work on hosted compilation will result in a lot of very useful tools and techniques.\nYou can also see in this section that I was also playing around with the symbolic operator \u0026gt;\u0026gt;= to compose together all the disposable streams at once. I suppose the inspiration for this (If you can call it that) is Reactive Extensions which has a CompositeDisposable, and also the disposable computation builder that Tomas Petricek made available as fssnip. The object expression that I used here seemed like a sensible option, and also shows the usefulness of object expressions.\nFinally we implement the interface using the FSharpEngine as the type to be stored in the scripting session:\ntype FSharpScriptEngine( scriptHostFactory:IScriptHostFactory, logger: ILog) = let mutable baseDir = String.empty let [\u0026lt;Literal\u0026gt;]sessionKey = \u0026#34;F# Session\u0026#34; interface IScriptEngine with member x.BaseDirectory with get() = baseDir and set value = baseDir \u0026lt;- value member x.Execute(code, args, references, namespaces, scriptPackSession) = let distinctReferences = references.Union(scriptPackSession.References).Distinct() let sessionState = match scriptPackSession.State.TryGetValue sessionKey with | false, _ -\u0026gt; let host = scriptHostFactory.CreateScriptHost(ScriptPackManager(scriptPackSession.Contexts), args) logger.Debug(\u0026#34;Creating session\u0026#34;) let session = new FSharpEngine(host) distinctReferences |\u0026gt; Seq.iter (fun ref -\u0026gt; logger.DebugFormat(\u0026#34;Adding reference to {0}\u0026#34;, ref) session.SilentAddReference ref ) namespaces.Union(scriptPackSession.Namespaces).Distinct() |\u0026gt; Seq.iter (fun ns -\u0026gt; logger.DebugFormat(\u0026#34;Importing namespace {0}\u0026#34;, ns) session.SilentImportNamespace ns) let sessionState = SessionState\u0026lt;_\u0026gt;(References = distinctReferences, Session = session) scriptPackSession.State.Add(sessionKey, sessionState) sessionState | true, res -\u0026gt; logger.Debug(\u0026#34;Reusing existing session\u0026#34;) let sessionState = res :?\u0026gt; SessionState\u0026lt;FSharpEngine\u0026gt; let newReferences = match sessionState.References with | null -\u0026gt; distinctReferences | refs when Seq.isEmpty refs -\u0026gt; distinctReferences | refs -\u0026gt; distinctReferences.Except refs newReferences |\u0026gt; Seq.iter (fun ref -\u0026gt; logger.DebugFormat(\u0026#34;Adding reference to {0}\u0026#34;, ref) sessionState.Session.AddReference ref ) sessionState match sessionState.Session.Execute(code) with | Success result -\u0026gt; let cleaned = result.Split([|\u0026#34;\\r\u0026#34;; \u0026#34;\\n\u0026#34;|], StringSplitOptions.RemoveEmptyEntries) |\u0026gt; Array.filter (fun str -\u0026gt; not(str = \u0026#34;\u0026gt; \u0026#34;)) |\u0026gt; String.concat \u0026#34;\\r\\n\u0026#34; ScriptResult(ReturnValue = cleaned) | Error e -\u0026gt; ScriptResult(CompileException = exn e ) | Incomplete -\u0026gt; ScriptResult() For the most part Execute is a simple port of the Roslyn implementation, mainly due to the way ScriptCs is currently implemented. There is a preprocessor that amongst other things parses reference additions (#r), passing them down to the Execute method. I think eventually a registrable command plugin for ScriptCs will appear that will make custom REPL commands easy to add and configure.\nAny new references are added in this snippet, where we leverage pattern matching.\nlet newReferences = match sessionState.References with | null -\u0026gt; distinctReferences | refs when Seq.isEmpty refs -\u0026gt; distinctReferences | refs -\u0026gt; distinctReferences.Except refs newReferences |\u0026gt; Seq.iter (fun ref -\u0026gt; logger.DebugFormat(\u0026#34;Adding reference to {0}\u0026#34;, ref) sessionState.Session.AddReference ref ) Also of note is the final matching block match sessionState.Session.Execute(code) with we use pattern matching against the discriminated union that is returned by Session.Execute(code). If Execute returns a Success we do a bit of a clean up on the result. We split the string based on carriage returns and newlines, filter out any prompts \u0026gt; , then reassemble the sting using String.Concat. We put this into the Result property of a ScriptResult. I do actually have a version of FsiEvaluationSession that suppresses prompts but I\u0026rsquo;ve not merged that in yet. An Error results in the CompileException property being used on the ScriptResult. Finally if the expression is Incomplete we don\u0026rsquo;t output a result or display an error as we are waiting for more input, we simply return an empty ScriptResult.\nmatch sessionState.Session.Execute(code) with | Success result -\u0026gt; let cleaned = result.Split([|\u0026#34;\\r\u0026#34;; \u0026#34;\\n\u0026#34;|], StringSplitOptions.RemoveEmptyEntries) |\u0026gt; Array.filter (fun str -\u0026gt; not(str = \u0026#34;\u0026gt; \u0026#34;)) |\u0026gt; String.concat \u0026#34;\\r\\n\u0026#34; ScriptResult(ReturnValue = cleaned) | Error e -\u0026gt; ScriptResult(CompileException = exn e ) | Incomplete -\u0026gt; ScriptResult() The final part is to plug this into ScriptSC, we do this by changing the Initialize method of CompositionRoot, all we need to do is Register our engine rather than the RoslynScriptEngine one:\nbuilder.RegisterType\u0026lt;ScriptExecutor\u0026gt;().As\u0026lt;IScriptExecutor\u0026gt;(); builder.RegisterType\u0026lt;RoslynScriptEngine\u0026gt;().As\u0026lt;IScriptEngine\u0026gt;(); To this:\nbuilder.RegisterType\u0026lt;ScriptExecutor\u0026gt;().As\u0026lt;IScriptExecutor\u0026gt;(); builder.RegisterType\u0026lt;FSharpScriptEngine\u0026gt;().As\u0026lt;IScriptEngine\u0026gt;(); And there we have it, hack session complete!\nSeveral thing are missing from my implementation, namely debug support and script pack support. The guys over at ScriptCs are continuing to evolve the API to allow plugins like this to work properly. Multi-line support should be coming soon, if you run a REPL session using F# then a prompt is added when you hit return. There is also a GitHub issue raised to add runtime packs for plugging in different runtimes/languages, this will pave the way for ScriptCs to be available on Mono too (If you disable the Roslyn based project and only use the F# Engine then it does actually work on Mono now.).\nOnce these issues are resolved in ScriptCs then hopefully F# will become a simple language plugin or even be merged into ScriptCs itself.\nYou can find my GitHub repository here, feel free to hack away, add issues, pull requests are welcome too!\nUntil next time!\n","date":"June 21, 2013","externalUrl":null,"permalink":"/programming/2013-06-21-can-i-have-some-fsharp-with-that/","section":"Blog","summary":" There’s been a fair bit of activity lately from a project called ScriptCS, it allows you to put together a project using C# as a lightweight scripting language, forgoing the use of Visual Studio which can sometimes be too bloated and bulky.\n","title":"Can I have some F# with that?","type":"programming"},{"content":"","date":"June 5, 2013","externalUrl":null,"permalink":"/tags/async/","section":"Tags","summary":"","title":"Async","type":"tags"},{"content":"","date":"June 5, 2013","externalUrl":null,"permalink":"/tags/mono/","section":"Tags","summary":"","title":"Mono","type":"tags"},{"content":" This creature is capable of tremendous destruction due to it\u0026rsquo;s size, flight (with the creature\u0026rsquo;s wings also generating hurricane strength winds) and possesses several breath weapons (e.g., heat and energy).\nWhat am I talking about here? Maybe it\u0026rsquo;s Monster Zero or King Ghidorah as it\u0026rsquo;s sometimes known. No it\u0026rsquo;s TPL Dataflow! Yeah, yeah, I have a penchant for being over dramatic and writing quirky intros. This post is about TPL Dataflow otherwise known as TDF. I have blogged about this before in my TDF agent series but I thought it might be worth while returning to it while on the subject of monsters.\nThe purpose of these posts is not to put you of using these libraries but to give you feel of how they might be used from an F# viewpoint. Its not always easy to use these libraries even from C#, so jumping to another paradigm can sometimes leave you feeling bewildered, frustrated and lost.\nWhat is TPL Dataflow # According to MSDN here\u0026rsquo;s the description of TPL dataflow:\nThe Task Parallel Library (TPL) provides dataflow components to help increase the robustness of concurrency-enabled applications. These dataflow components are collectively referred to as the TPL Dataflow Library. This dataflow model promotes actor-based programming by providing in-process message passing for coarse-grained dataflow and pipelining tasks.\nCheckout the MSDN if you want to read a more in depth outline on TDF, or you could also refer to my earlier posts too.\nSample problem # Here\u0026rsquo;s a sample problem: We have two documents that we want to use to collate a list of word occurrences, we then want to collect the results from both documents and print out the combined results.\nAlthough we\u0026rsquo;re using TDF to solve this problem, we could of also used F# agents, Reactive Extensions, Linq, TPL, or a mix of all of those.\nWe will stick to using the F# REPL for this post as it\u0026rsquo;s fairly simple example and allows for a bit of interactivity. Lets start by adding a reference, open up a few namespace\u0026rsquo;s and read a couple of text files in. In this instance we\u0026rsquo;re going to use Jane Eyre by Charlotte Bronte, and The Wendigo by Algernon Blackwood, for no reason other than they were free to download and use as samples.\n#r \u0026#34;System.Threading.Tasks.Dataflow\u0026#34; open System open System.IO open System.Threading.Tasks.Dataflow let janeEyre = File.ReadAllText(@\u0026#34;Jane Eyre [Charlotte Bronte].txt\u0026#34;) let theWendigo = File.ReadAllText(@\u0026#34;The Wendigo [Algernon Blackwood].txt\u0026#34;) The next thing we need to think about is how to count the words. Let\u0026rsquo;s create a recursive function that counts each occurrence of a passed in word against the full text. The wordCount function recurses until -1 is returned from the IndexOf function. Before you argue about memory allocation, laziness etc, we\u0026rsquo;re not interested in that at the moment, we just want to solve the problem at hand. It would be fairly easy to split the input into a lazy sequence and iterate over it so we only keep a single line in memory.\nlet wordCount (text: String) word = let rec loop position count = match text.IndexOf(word, position, StringComparison.InvariantCultureIgnoreCase) with | -1 -\u0026gt; count | i -\u0026gt; loop (i + word.Length) (count + 1) loop 0 0 Now we can start to create some TDF blocks, we shall create a BroadcastBlock first.\nA BroadcastBlock provides a buffer for storing at most one element at time, overwriting each message with the next as it arrives.\nMessages are broadcast to all linked targets, all of which may consume a clone of the message.\nThe lambda expression passed into the BroadcastBlock is it\u0026rsquo;s clone function, in this case we will just use the string that is passed in: fun s -\u0026gt; s. We will be using the BroadcastBlock to send the same message to multiple destinations later on.\nlet broadcast = BroadcastBlock(fun s -\u0026gt; s) Now we will create a couple of TransformBlocks.\nA TransformBlock provides a dataflow block that invokes a provided Func(T, TResult) delegate for every data element received.\nThe TransformBlock will accept a data element and transform it by invoking it\u0026rsquo;s transform function. Here we will partially apply the wordCount function by passing in the first parameter text. This means that whenever the TransformBlock is passed data the wordCount function will already have the text parameter applied and will return the number of matches from the document. The context of passed data here will be the word that we want to find. We\u0026rsquo;ll create one for Jane Eyre and one for The Wendigo:\nlet transformJaneEyre = TransformBlock(wordCount janeEyre) let transformTheWendigo = TransformBlock(wordCount theWendigo) Now that we have created three blocks we need to think about linking them together. You can do this with the LinkTo method that every dataflow block has. We could do that by calling the LinkTo methods as usual like this:\nbroadcast.LinkTo(transformJaneEyre) |\u0026gt; ignore broadcast.LinkTo(transformTheWendigo) |\u0026gt; ignore That seems a little awkward, especially as we\u0026rsquo;re not interested in the return parameter in this example, so instead we\u0026rsquo;re going to create a little function to make this a little bit easier for ourselves:\nlet (--\u0026gt;) source target = DataflowBlock.LinkTo(source, target) |\u0026gt; ignore The LinkTo method returns an IDisposable that can be use to sever the link between the blocks that have just been joined. There are also overloads of LinkTo that allow you to specify a predicate. There is also an overload that takes a DataflowLinkOptions type which allows you to specify whether the new link is appended (Append), the maximum message that can be passed before the block is unlinked (MaxMessages), and finally and whether or not the completion of the former block is propagated to the latter (PropagateCompletion).\nWe can now use the --\u0026gt; symbolic operator and use infix notation to link the blocks together. Infix operators are expected to be placed between the two operands, which means we can define the links between the block like this:\nbroadcast --\u0026gt; transformJaneEyre broadcast --\u0026gt; transformTheWendigo Next we create a JoinBlock.\nA JoinBlock provides a dataflow block that joins across multiple dataflow sources, which are not necessarily of the same type, waiting for one item to arrive for each type before they’re all released together as a tuple that contains one item per type.\nlet join = JoinBlock\u0026lt;_,_,_\u0026gt;() broadcast --\u0026gt; join.Target1 transformJaneEyre --\u0026gt; join.Target2 transformTheWendigo --\u0026gt; join.Target3 We create the JoinBlock then link the broadcast, transformJaneEyre, and transformTheWendigo to it. This means that the JoinBlock will wait for data from all three blocks before sending the data on as a tuple of the three values.\nFinally we create the last block which is an ActionBlock\nAn ActionBlock provides a dataflow block that invokes a provided Action(T) delegate for every data element received.\nlet writeOutput = ActionBlock(fun(word:String, count1, count2) -\u0026gt; Console.WriteLine(\u0026#34;Word: {0}, Jane Eyre: {1}, The Wendigo: {2}\u0026#34;, word.PadRight(10), (string count1).PadLeft(3), (string count2).PadLeft(3) ) ) join --\u0026gt; writeOutput Right, that completes the hook up, now all that\u0026rsquo;s left is to test it.\nlet words = [|\u0026#34;cat\u0026#34;;\u0026#34;cake\u0026#34;;\u0026#34;anything\u0026#34;;\u0026#34;laugh\u0026#34;;\u0026#34;breeze\u0026#34;;\u0026#34;hysterical\u0026#34;;\u0026#34;ball\u0026#34;;\u0026#34;them\u0026#34;;\u0026#34;home\u0026#34;;\u0026#34;bird\u0026#34;|] for word in words do broadcast.Post(word) |\u0026gt; ignore If we execute this then we we get the following output:\nWord: cat , Jane Eyre: 212, The Wendigo: 73 Word: cake , Jane Eyre: 15, The Wendigo: 0 Word: anything , Jane Eyre: 60, The Wendigo: 19 Word: laugh , Jane Eyre: 68, The Wendigo: 17 Word: breeze , Jane Eyre: 11, The Wendigo: 0 Word: hysterical, Jane Eyre: 1, The Wendigo: 1 Word: ball , Jane Eyre: 11, The Wendigo: 1 Word: them , Jane Eyre: 432, The Wendigo: 72 Word: home , Jane Eyre: 90, The Wendigo: 11 Word: bird , Jane Eyre: 35, The Wendigo: 0 Summary # From a conceptual viewpoint of view we\u0026rsquo;re creating a BroadcastBlock which connects to two TransformBlocks. The two TransformBlocks are then connected to the JoinBlock along with the BroadcastBlock. Finally, the JoinBlock is connected to the ActionBlock.\nThis creates a mini network where you can simply post a message to the input of the dataflow network and it will propagate through the network. As with Reactive Extensions marble diagrams and pipeline diagrams are a great way to visualise the process flow.\nI hope that sheds a little bit of light on how a dataflow network can be created with TDF. Extremely complex behaviour\u0026rsquo;s can be created by connecting up the simple dataflow building blocks, especially as the different block\u0026rsquo;s can run with multiple degrees of parallelism an also run asynchronously using Task\u0026lt;T\u0026gt;.\nUntil next time!\n","date":"June 5, 2013","externalUrl":null,"permalink":"/programming/2013-06-05-monster-zero-revisited/","section":"Blog","summary":" This creature is capable of tremendous destruction due to it’s size, flight (with the creature’s wings also generating hurricane strength winds) and possesses several breath weapons (e.g., heat and energy).\nWhat am I talking about here? Maybe it’s Monster Zero or King Ghidorah as it’s sometimes known. No it’s TPL Dataflow! ","title":"Monster Zero - Revisited","type":"programming"},{"content":"","date":"June 5, 2013","externalUrl":null,"permalink":"/tags/tpl-dataflow/","section":"Tags","summary":"","title":"Tpl-Dataflow","type":"tags"},{"content":" What\u0026rsquo;s 100 meters high and weighs in at around 60,000 tons? No its not Godzilla, its Reactive extensions!\nLately on one of my projects I have been doing a lot of stream manipulation, and although I solved the problem quite easily using F# async workflows, there were other solutions available to help solve the problem. I could of used things like async await, TPL Dataflow(TDF), and Reactive Extensions (Rx). This is going to be a short post on using Rx with F#. What is Rx? # Well for those of you that don\u0026rsquo;t know anything about Rx I would suggest reading up a bit of the introduction material here. Here\u0026rsquo;s a quick recap of what Rx is for everyone else:\nThe Reactive Extensions (Rx) is a library for composing asynchronous and event-based programs using observable sequences and LINQ-style query operators. Using Rx, developers represent asynchronous data streams with Observables , query asynchronous data streams using LINQ operators , and parameterize the concurrency in the asynchronous data streams using Schedulers . Simply put, Rx = Observables + LINQ + Schedulers.\nSample problem # Here\u0026rsquo;s a sample problem: I want to read from a file in buffered chunks and perform an action whenever a chunk is read. As I said before there a many different ways to solve this problem but We\u0026rsquo;ll use C# as a base to see how it could be done first in C#. For this example We\u0026rsquo;ll add an Extension extension method to the Stream class:\npublic static class Extensions { public static IObservable\u0026lt;byte[]\u0026gt; ToObservable (this Stream stream, int size) { return Observable.Create\u0026lt;byte[]\u0026gt; (observer =\u0026gt; { byte[] buffer = new byte[size]; var deferedRead = Observable.Defer(() =\u0026gt; stream.ReadAsync (buffer, 0, size).ToObservable()); return Observable.Repeat(deferedRead) .Select (i =\u0026gt; buffer.Take(i).ToArray ()) .Subscribe (data =\u0026gt; { if (data.Length \u0026gt; 0) observer.OnNext (data); else observer.OnCompleted (); }, observer.OnError, observer.OnCompleted); }); } } Using this extension method we can then do something like this:\nvar source = new FileStream (@\u0026#34;test.txt\u0026#34;, FileMode.Open, FileAccess.Read); source.ToObservable (16).Subscribe (_ =\u0026gt; Console.WriteLine(_.Length)); This will print to the console the length of each chunk as it is read.\nThere are quite a few different Rx operators in this example, Create, Defer, ToObservable, Repeat, Select, and Subscribe. Lets quickly go though the example and see what\u0026rsquo;s going on.\nFirst of all we create a custom observable sequence using Observable.Create. This takes a lambda function with a single parameter observer, which is of type IObserver\u0026lt;byte[]\u0026gt;. Using the observer we can produce elements in the sequence by using the methods OnNext, OnError and OnCompleted.\nNext up we create a buffer to hold the data which will be read from the file in chunks. This is just a simple array allocation byte[] buffer = new byte[size];\nTo allow us to consume the data from the file stream we can use the ReadAsync method which will return a Task\u0026lt;byte[]\u0026gt;. There is an Rx extension method on Task called ToObservable so we use that too. You will notice in the code that we are using Observable.Defer. Why are we using that? What would happen if we didn\u0026rsquo;t? Well, if we don\u0026rsquo;t defer the Task for later execution and simply use to Task.ToObservable() we would be creating a new instance of the Observable sequence each time ReadAsync is called - This would mean we would have an infinite sequence comprised of the first chunk of the file, which isnt waht we want at all. By using Defer we don\u0026rsquo;t invoke the Observable Task until first subscription to the Observable sequence.\nWe use a fluent style to repeat the deferred Observable deferedRead using the Repeat method.\nSelect is now used to take the number of bytes from the buffer, we might have a stream which is not divisible by the buffer size which will mean that the last read will not be the size of the buffer. In the lambda expression the i parameter is the number of the bytes returned from ReadAsync.\nFinally we have Subscribe, this takes a lambda that is passed data, data being the current chunk or byte[]. In the body of the lambda we check to see if we have received any bytes, if so then we call observer.OnNext(data) which creates the next element in the sequence. If we didn\u0026rsquo;t receive data then we call observer.OnCompleted(), which completes the sequence. The last two parameters for Subscribe are the error and completed actions, we simply use the ones in the observer - observer.OnError and observer.OnCompleted.\nIf you read through the code again now it probably makes more sense the second time around, there\u0026rsquo;s a lot of functionality squeezed into a small space but by using tried and tested components / functions in Rx you should have a better experience than rolling your own parts, of course you can do the same with TDF but I wont go into that here.\nSo what would all this look like in F#?\nWell, if you try to do a direct port you start to hit a few issues due to the amount of overloads for some of the methods, Zip for example, has a staggering 19 overloads!! This almost always means your working right at the edge of the ability of type inferencing. In order to determine what method overload you intended to use you have to add further type parameters, this can sometimes be a tricky business as F# lambda\u0026rsquo;s are not always correctly typed back to Action and Func.\nLets see an example of that now:\nmodule StreamExt = type Stream with member x.ToObservable(size) = Observable.Create(fun (observer: IObserver\u0026lt;_\u0026gt;) -\u0026gt; let buffer = Array.zeroCreate size let defered = Observable.Defer(fun () -\u0026gt; (x.ReadAsync (buffer, 0, size)).ToObservable()) Observable.Repeat\u0026lt;int\u0026gt;(defered) .Select(fun i -\u0026gt; buffer.Take(i).ToArray()) .Subscribe( (fun (data:byte[]) -\u0026gt; if data.Length \u0026gt; 0 then observer.OnNext(data) else observer.OnCompleted()), observer.OnError, observer.OnCompleted )) I had to add quite a few type annotations to get this working. You can end up spending quite a while adding explicit types which isn\u0026rsquo;t exactly an enjoyable or productive way of spending your time, sometimes you can hit a wall and have to annotate the function separately to see where the inference is failing.\nTo make things easier you can wrap the overloads with F# friendly versions. In fact this has already been done in the Fsharp.Reactive repo on GitHub. As I\u0026rsquo;m using Mono I had to do a quick compilation against the Reactive Extensions that come bundled with Mono 3.x rather than the nuget references. I also added in a couple of function\u0026rsquo;s that were missing from this version. Here\u0026rsquo;s the result usin FSharp.Reactive, I think you\u0026rsquo;ll agree it looks a bit better and seems to flow quite nice with the pipeline operators in place.\nmodule StreamExt = type Stream with member x.ToObservable(size) = Observable.Create (fun (observer: IObserver\u0026lt;_\u0026gt;) -\u0026gt; let buffer = Array.zeroCreate size Observable.Defer(fun () -\u0026gt; (x.ReadAsync (buffer, 0, size)).ToObservable()) |\u0026gt; Observable.repeat |\u0026gt; Observable.map(fun i -\u0026gt; buffer |\u0026gt; Seq.take i |\u0026gt; Seq.toArray) |\u0026gt; Observable.subscribe(function | data when data.Length \u0026gt; 0 -\u0026gt; observer.OnNext(data) | _ -\u0026gt; observer.OnCompleted()) observer.OnError observer.OnCompleted) Notable difference are the Observable.Defer is piped into repeat, map and subscribe. Finally, the last piece that\u0026rsquo;s different is the use of the Seq expression (fun i -\u0026gt; buffer |\u0026gt; Seq.take i |\u0026gt; Seq.toArray) rather than the Linq Take function .Select (i =\u0026gt; buffer.Take(i).ToArray ()). To be honest there\u0026rsquo;s not really much difference between the two, sequence expressions just seem more natural while using F#. Lastly I switched from the if else expression to a pattern matching using the function keyword. It\u0026rsquo;s used in pattern matching when we want to match against only one parameter that\u0026rsquo;s passed into the function. This makes the subscribe function a little more compact.\nHere is the details of the functions that were used in the above snippet so that you don\u0026rsquo;t have to go looking in GitHub for details:\nmodule Observable = ///Repeats the observable let repeat f = Observable.Repeat(source = f) /// maps the given observable with the given function let map f source = Observable.Select(source, Func\u0026lt;_,_\u0026gt;(f)) /// Subscribes to the observable with all three callbacks let subscribe onNext onError onCompleted (observable: \u0026#39;a IObservable) = observable.Subscribe(Observer.Create(Action\u0026lt;_\u0026gt; onNext, Action\u0026lt;_\u0026gt; onError, Action onCompleted)) I think that Rx is a very useful library but it\u0026rsquo;s ironic that a functional programming oriented library is not easily usable from a functional language like F#. There are over 400 Observable extension methods if you include all the overloads. Its like ten thousand spoons when all you need is a knife! \u0026hellip; Joking aside I wish the API designers had taken it easy when adding all the extension methods, when you are developing code the last thing you want to do is scroll up and down through method overloads trying to spot which exact overload you are looking for.\nIf you want some more samples you might want to take a look at 101 Rx Samples. Also for reference when building something new, make sure you check out the Reactive Extensions design Guide.\nUntil next time!\n","date":"June 1, 2013","externalUrl":null,"permalink":"/programming/2013-06-01-some-kind-of-monster/","section":"Blog","summary":" What’s 100 meters high and weighs in at around 60,000 tons? No its not Godzilla, its Reactive extensions!\nLately on one of my projects I have been doing a lot of stream manipulation, and although I solved the problem quite easily using F# async workflows, there were other solutions available to help solve the problem. I could of used things like async await, TPL Dataflow(TDF), and Reactive Extensions (Rx). This is going to be a short post on using Rx with F#. ","title":"Some kind of monster","type":"programming"},{"content":" What is Edge.js? # Unless you live in a hole you have probably heard of node.js so I\u0026rsquo;ll not bother to explain what it is or what it does. An interesting project has come to light lately, namely Edge.js. The Edge.js project allows you to connect node.js with .Net.\nThe creator of Edge.js Tomasz Janczuk sums this up nicely:\nAn edge connects two nodes\nThis edge connects node.js with .NET\nCurrently Edge.js is only available on Windows but there is work underway to bring this to Mono, thus opening up the possibilities even further. The coding model for Edge.js offers different integration options depending on the quantity of code you are writing, and whether you want to call a .Net dll directly.\nHere are a few examples:\nSingle line lambda expressions: # var edge = require(\u0026#39;edge\u0026#39;); var hello = edge.func( \u0026#39;async (input) =\u0026gt; { return \u0026#34;.NET welcomes \u0026#34; + input.ToString(); }\u0026#39; ); hello(\u0026#39;Node.js\u0026#39;, function (error, result) { if (error) throw error; console.log(result); }); Multi line lambda expressions: # var hello = require(\u0026#39;edge\u0026#39;).func(function () {/* async (input) =\u0026gt; { return \u0026#34;.NET welcomes \u0026#34; + input.ToString(); } */}); hello(\u0026#39;Node.js\u0026#39;, function (error, result) { ... }); File based expresions: # var hello = require(\u0026#39;edge\u0026#39;).func(\u0026#39;hello.csx\u0026#39;); hello(\u0026#39;Node.js\u0026#39;, function (error, result) { ... }); Invoking via a dll: # var add7 = require(\u0026#39;edge\u0026#39;).func(\u0026#39;My.Sample.dll\u0026#39;); add7(12, function (error, result) { ... } The entry point into your .NET code is a delegate normalized to a Func\u0026lt;object,Task\u0026lt;object\u0026gt;\u0026gt;. This allows node.js code to call the .NET code asynchronously and avoid blocking the node.js event loop. If you think about the possibilities of this for a moment, a lot of different options begin to open up with this framework. I can foresee a lot of interesting things appearing in the future.\nThere are currently two .Net compilers part of Edge.js. A C# based compiler and an IronPython one. You can probably guess what I\u0026rsquo;m going say next\u0026hellip;\nIntroducing Edge-fs - An F# complier for edge.js # First let\u0026rsquo;s look at the interop model for Edge.js: In summary, if we want to integrate with Edge.js then we must coerce whatever input that is passed to a single delegate function Func\u0026lt;Object, Task\u0026lt;Object\u0026gt;\u0026gt;\nIn terms of the C# Edge compiler a lambda expression is passed in the async await style:-\nasync (input) =\u0026gt; { return \u0026#34;.NET welcomes \u0026#34; + input.ToString(); } The Python Edge compiler is passed a lambda in it\u0026rsquo;s native format too:\ndef hello(input): return \u0026#34;Python welcomes \u0026#34; + input lambda x: hello(x) So where does that leave us with F# compiler support? Well, I suppose the most intuitive support for F# would be to use F# async workflow support. This would mean the that lambda expression would look like this:\nfun input -\u0026gt; async{return \u0026#34;.NET welcomes \u0026#34; + input.ToString()} You can see it\u0026rsquo;s not that different from C#\u0026rsquo;s\u0026rsquo; async await style syntax, you can really see the F# async workflow heritage here.\nScript Example # Now lets look at how a script file or dll and have a look to see how this would fits:\nnamespace global type Startup() = let addSeven v = v + 7 member x.Invoke(input:obj) = let v = input :?\u0026gt; int async.Return (addSeven v :\u0026gt; obj) |\u0026gt; Async.StartAsTask This is really easy too, the Async module has a StartAsTask function that perfectly fits here.\nBy default Edge.js looks for a type in the global namespace called Startup with a public method called Invoke. The invoke method takes a single parameter input which is of the type Object. The return type of this method is as you might have guessed Task\u0026lt;Object\u0026gt;. You can also add parameters to the node.js to indicate the location of the assembly, type and method name using the assemblyName, typeName and methodName parameters respectively.\nvar clrMethod = edge.func({ assemblyFile: \u0026#39;My.Edge.Samples.dll\u0026#39;, typeName: \u0026#39;Samples.FooBar.MyType\u0026#39;, methodName: \u0026#39;MyMethod\u0026#39; }); Further Documentation # Edge.js has some really good documentation so if your interested then you really should check it out. I plan on supporting all of the calling conventions that the C# edge compiler has to offer. At the moment only the in-line lambdas and the file based inputs have been tested, but I\u0026rsquo;m working on further examples, and fixes as needed.\nWhy do you need a Custom Compiler # As an aside, with dll based inputs any .Net language would work with Edge.js, you don\u0026rsquo;t need a custom compiler. The internals of Edge.js invoke your dll via reflection.\nHandle\u0026lt;v8::Value\u0026gt; ClrFunc::Initialize(const v8::Arguments\u0026amp; args) { ... // reference .NET code through pre-compiled CLR assembly String::Utf8Value assemblyFile(jsassemblyFile); String::Utf8Value nativeTypeName(options-\u0026gt;Get(String::NewSymbol(\u0026#34;typeName\u0026#34;))); String::Utf8Value nativeMethodName(options-\u0026gt;Get(String::NewSymbol(\u0026#34;methodName\u0026#34;))); typeName = gcnew System::String(*nativeTypeName); methodName = gcnew System::String(*nativeMethodName); assembly = Assembly::LoadFrom(gcnew System::String(*assemblyFile)); ClrFuncReflectionWrap^ wrap = ClrFuncReflectionWrap::Create(assembly, typeName, methodName); result = ClrFunc::Initialize( gcnew System::Func\u0026lt;System::Object^,Task\u0026lt;System::Object^\u0026gt;^\u0026gt;( wrap, \u0026amp;ClrFuncReflectionWrap::Call)); ... A custom compiler is only required for compiling code in the form of scripts or lambda expressions. It\u0026rsquo;s expected that this will be a common use case so it\u0026rsquo;s important to have a native F# compiler support.\nSo there we have it, a very quick whistle stop tour of Edge-fs the F# compiler for Edge.js. I realise that this post only just skims the surface but I just wanted to get this out in the wild. Ill be updating my repo over the next day or so, and a stable release will go out via the npm package as soon as things stabilise.\nNext time we\u0026rsquo;re going to lift the lid on the F# Edge compiler and take a look at it\u0026rsquo;s guts, we\u0026rsquo;ll also go through some of the trials and tribulations I had along the way. Ill also continue the series with some more documentation and samples too.\nUntil next time!\n","date":"May 5, 2013","externalUrl":null,"permalink":"/programming/2013-05-05-i-node-something/","section":"Blog","summary":"What is Edge.js? # Unless you live in a hole you have probably heard of node.js so I’ll not bother to explain what it is or what it does. An interesting project has come to light lately, namely Edge.js. The Edge.js project allows you to connect node.js with .Net.\n","title":"I node something (Bout You)","type":"programming"},{"content":"In this post weare going to look as async again, but from the perspective of F#.\nXamarin Evolve 2013 # I have been watching the Xamarin Evolve conference this week and it was good to see Miguel announce full support for F#. Those that follow me on twitter etc, will know that I have been doing F# for quite a while in MonoDevelop and Xamarin Studio. The new support currently entails some new project templates so that you can easily create epic new F# Apps without having to refer to my blog. While its sad that my content now falls into the archives its nice to get official support announced in such a grand fashion. F# Async # Kudos to Miguel for covering some history of C#\u0026rsquo;s async feature right back down to its F# heritage too, which appeared in 2007, thanks to the work of Don Syme and the F# team. You can read more about that on Don Syme\u0026rsquo;s blog or have a look at the research paper here.\nSo the highly anticipated async await model in C# that\u0026rsquo;s just gone beta in Xamarin addin channel? We\u0026rsquo;ve had it for ages in F#! In fact, I suspect you will have been able to use it for quite some time, even before I started hacking together support for F# in iOS! Anyway, that\u0026rsquo;s enough of the smugness :-) lets get on and see what it looks like using some of the code from the previous post as a reference.\nIll include the C# version first so that you can see the difference rather than having to open my last post.\n// Asynchronous HTTP request public async void HttpSample () { Application.Busy (); var request = WebRequest.Create (Application.WisdomUrl); //async await version try{ var response = await request.GetResponseAsync(); Application.Done (); ad.RenderRssStream(response.GetResponseStream()); } catch { // Error } } One of the advantages of the F# Async model is it\u0026rsquo;s composable nature and controllability. The key to F# async is that its defined with F#\u0026rsquo;s computation expression syntax:\nComputation expressions in F# provide a convenient syntax for writing computations that can be sequenced and combined using control flow constructs and bindings.\nThere are several built in workflows: Sequences, Asynchronous Workflows, and Query Expressions. Whenever you use a computation expression it is as follows:- builder-name { expression }. With that tiny bit of background, lets look at the corresponding F# async code:\nmember x.HttpSample() = Application.Busy() let request = WebRequest.Create(Application.WisdomUrl ) //F# async version async {try let! response = request.AsyncGetResponse() Application.Done() ad(response.GetResponseStream()) with ex -\u0026gt; () } |\u0026gt; Async.Start You can see there is quite a similarity between this snippet and the C# one and you should be able to figure out what\u0026rsquo;s happening given the knowledge from the previous post.\nOne of the first things you will notice the builder - async { ..., followed by the let! statement. You can think of the let! as the C# equivalent of await. let! starts the computation request.AsyncGetResponse(), and then the thread is suspended until the result is available, at this point execution continues to the next statment, which in this case is Application.Done().\nThose of you comparing the difference will notice that in the C# version after Application.Done(); we call ad.RenderRssStream(response.GetResponseStream()) but in the F# version we simply call ad(response.GetResponseStream()). If we take a quick look at the constructors for the types that hold these methods I can show you the difference a bit better:\nThe C# version looks like this:\npublic class DotNet { AppDelegate ad; public DotNet (AppDelegate ad) { this.ad = ad; } } The F# one I can show on a single line:\ntype DotNet(ad: Stream -\u0026gt; unit) = The main difference is that The C# version has the entire AppDelegate class is passed in, whereas the F# version just takes a function with the signature Stream -\u0026gt; unit. In fact the F# version doesn\u0026rsquo;t even need to be placed inside a type like the C# version, we can use a module, again Ill quote from MSDN:\nIn the context of the F# language, a module is a grouping of F# code, such as values, types, and function values, in an F# program. Grouping code in modules helps keep related code together and helps avoid name conflicts in your program.\nF# Modules # module DotNet\u0026#39; = let HttpSample(ad) = Application.Busy() let request = WebRequest.Create(Application.WisdomUrl ) //F# async version async {try let! response = request.AsyncGetResponse() Application.Done() ad(response.GetResponseStream()) with ex -\u0026gt; () } |\u0026gt; Async.Start When we want to call this code we can open the module like you would a namespace:\nopen DotNet HttpSample(ad) Or access it fully qualified by including the module name:\nDotNet.HttpSample(ad) How would this code look from the context of this sample application?\nHere is a snipped from the AppDelegate code which makes use of this module\n// This method is invoked when the application has loaded its UI and its ready to run override x.FinishedLaunching (app:UIApplication, options:NSDictionary) = x.window.AddSubview (x.navigationController.View) x.button1.TouchDown.Add (fun _ -\u0026gt; if not UIApplication.SharedApplication.NetworkActivityIndicatorVisible then match x.stack.SelectedRow() with | 0 -\u0026gt; DotNet.HttpSample x.RenderRssStream | 1 -\u0026gt; DotNet.HttpSecureSample x.RenderStream | _ -\u0026gt; (new Cocoa(x.RenderRssStream)).HttpSample() |\u0026gt; ignore ) TableViewSelector.Configure (x.stack, [|\u0026#34;http - WebRequest\u0026#34; \u0026#34;https - WebRequest\u0026#34; \u0026#34;http - NSUrlConnection\u0026#34; |] ) x.window.MakeKeyAndVisible() true There are a few departures from the C# sample code which Ill include below now:\n// This method is invoked when the application has loaded its UI and its ready to run public override bool FinishedLaunching (UIApplication app, NSDictionary options) { window.AddSubview (navigationController.View); button1.TouchDown += Button1TouchDown; TableViewSelector.Configure (this.stack, new string [] { \u0026#34;http - WebRequest\u0026#34;, \u0026#34;https - WebRequest\u0026#34;, \u0026#34;http - NSUrlConnection\u0026#34; }); window.MakeKeyAndVisible (); return true; } void Button1TouchDown (object sender, EventArgs e) { // Do not queue more than one request if (UIApplication.SharedApplication.NetworkActivityIndicatorVisible) return; switch (stack.SelectedRow ()){ case 0: new DotNet (this).HttpSample (); break; case 1: new DotNet (this).HttpSecureSample (); break; case 2: new Cocoa (this).HttpSample (); break; } } Firstly we are using an lambda expression for the event handler via the Add method rather than the += handler which we use in C#. We are also using F#\u0026rsquo;s awesome pattern matching feature on the results of x.stack.SelectedRow(). This allows you to encode complex logic and also have the compiler assist you by catching non covered cases.\nI\u0026rsquo;m going to leave it there for now as I don\u0026rsquo;t want to bombard any newcomers with tons of new F# features, and I also don\u0026rsquo;t want to teach any of my regular F# followers how to suck eggs. If anyone has a preference for more in depth comparisons to the C# version then let me know then I can tailor that into further posts on the subject.\nUntil next time!\n","date":"April 18, 2013","externalUrl":null,"permalink":"/programming/2013-04-18-ios-async-revisited/","section":"Blog","summary":"In this post weare going to look as async again, but from the perspective of F#.\nXamarin Evolve 2013 # I have been watching the Xamarin Evolve conference this week and it was good to see Miguel announce full support for F#. Those that follow me on twitter etc, will know that I have been doing F# for quite a while in MonoDevelop and Xamarin Studio. The new support currently entails some new project templates so that you can easily create epic new F# Apps without having to refer to my blog. While its sad that my content now falls into the archives its nice to get official support announced in such a grand fashion. ","title":"iOS async revisited","type":"programming"},{"content":"I was going to title this post as \u0026lsquo;Now for something completely different\u0026rsquo; but felt that a little bit too Pythonesque, and when I thought about it a bit it isn\u0026rsquo;t really completely just slightly different, namely C# rather than my usual F# posts. Right, enough of the waffling, this post is a little tour into the relatively unknown area of async on iOS. Xamarin announced the alpha preview of async await on March 11th this year (2013). There are a couple of blog post floating around on the net if you look around, Rodrigo Kumpera posted a small example here.\nI\u0026rsquo;m no stranger to async, I have spent a great deal of time over the years debugging and refining IAsyncResult style procedures and found Jeffrey Richter and Joe Duffy\u0026rsquo;s books below to be an excellent reference for those interested.\nIf you looking for a book on TPL/parallel programming then Parallel Programming with Microsoft .NET by Stephen Toub et al. is also a good read.\nAnyway, enough of the book references lets look at an iOS example of async using the MonoTouch samples from Xamarin as a reference.\nWe are going to use the HttpSample. Its a relatively simple example that has several buttons which trigger an asynchronous request for data, when the data is returned it\u0026rsquo;s simply rendered onto the screen.\nLets look at the first asynchronous call in the DotNet.cs file, the HttpSample method:\n// Asynchronous HTTP request public void HttpSample () { Application.Busy (); var request = WebRequest.Create (Application.WisdomUrl); request.BeginGetResponse (FeedDownloaded, request); } // Invoked when we get the stream back from the twitter feed // We parse the RSS feed and push the data into a table. void FeedDownloaded (IAsyncResult result) { Application.Done (); var request = result.AsyncState as HttpWebRequest; try { var response = request.EndGetResponse (result); ad.RenderRssStream (response.GetResponseStream ()); } catch { // Error\t} } To convert this to the async await style all we have to do is use the async and await keywords (surprise surprise!), and in this instance use the new Async suffixed methods on the WebRequest class.\n// Asynchronous HTTP request public async void HttpSample () { Application.Busy (); var request = WebRequest.Create (Application.WisdomUrl); //async await version try{ var response = await request.GetResponseAsync(); Application.Done (); ad.RenderRssStream(response.GetResponseStream()); } catch { // Error } } As you can see the request.BeginGetResponse method has been changed to await request.GetResponseAsync() and the callback method FeedDownloaded which was passed into the BeginGetRespose method has now been assimilated into the HttpSample method. The await keyword is acting as a wait point or suspension while the asynchronous method completes. As soon the asynchronous call completes then processing continues to the line below, much in the same way that the callback code is executed in the IAsyncResult version. For an in depth description then you can take a look at the MSDN documentation on the subject.\nYou could add a WebException to the catch block here, you would expect on situations like network outage which would result in DNS lookup failures from the async call.\nWe can look at the next asynchronous method too the \u0026lsquo;HttpSecureSample\u0026rsquo; method\n// Asynchornous HTTPS request public void HttpSecureSample () { var https = (HttpWebRequest) WebRequest.Create (\u0026#34;https://gmail.com\u0026#34;); // To not depend on the root certficates, we will accept any certificates: ServicePointManager.ServerCertificateValidationCallback = (sender, cert, chain, ssl) =\u0026gt; true; https.BeginGetResponse (GmailDownloaded, https); } // This sample just gets the result from calling https://gmail.com, an HTTPS secure // connection, we do not attempt to parse the output, but merely dump it as text void GmailDownloaded (IAsyncResult result) { Application.Done (); var request = result.AsyncState as HttpWebRequest; try { var response = request.EndGetResponse (result); ad.RenderStream (response.GetResponseStream ()); } catch { // Error } } Lets have a look at the async await version of that:\n// Asynchornous HTTPS request public async void HttpSecureSample () { var https = (HttpWebRequest) WebRequest.Create (\u0026#34;https://gmail.com\u0026#34;); // To not depend on the root certficates, we will accept any certificates: ServicePointManager.ServerCertificateValidationCallback = (sender, cert, chain, ssl) =\u0026gt; true; try { var response = await https.GetResponseAsync(); Application.Done (); ad.RenderRssStream (response.GetResponseStream ()); } catch { // Error } } In this situation there was also a version of the IAsyncResult Begin/End pattern that had been converted to async - https.GetResponseAsync() and we also had to add the async keyword to the HttpSecureSample method.\nFor the situations where there is no Async suffixed method available you can build your own using the Task.Factory.FromAsync methods. I wont go into the details of that here but if anyone wants any information on that then just give me a shout and I can revisit in in a future post.\nAh yes, I almost forgot, there are some common pitfalls of using async await and Tomas Petricek posted a good compilation of them the other day: C# async gotchas.\nUntil next time!\n","date":"April 16, 2013","externalUrl":null,"permalink":"/programming/2013-04-16-a-little-bit-of-ios-async/","section":"Blog","summary":"I was going to title this post as ‘Now for something completely different’ but felt that a little bit too Pythonesque, and when I thought about it a bit it isn’t really completely just slightly different, namely C# rather than my usual F# posts. ","title":"A little bit of iOS async","type":"programming"},{"content":"","date":"February 7, 2013","externalUrl":null,"permalink":"/tags/monotouch/","section":"Tags","summary":"","title":"Monotouch","type":"tags"},{"content":"In the last post we left at the point where everything was running fine and dandy on the Simulator. So what happens if we compile for the real hardware?\nLets change the active configuration to Debug|iPhone and hit build, what do we get?\nBoom! # Error MT2002: Could not resolve: FSharp.Core, Version=4.3.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a (MT2002) (singleview)\nSo I guess we need to tell it where the FSharp.Core.dll is, lets add a reference to it:\n/Libraries/FrameWorks/Mono.FrameWork/Libraries/mono/Microsoft F#/v4.0/Fsharp.Core.dll Now try and build \u0026hellip; another weird error:\nunknown-file(1,1): Error FS2020: The assembly \u0026lsquo;F#/v4.0/FSharp.Core.dll\u0026rsquo; is listed on the command line. Assemblies should be referenced using a command line flag such as \u0026lsquo;-r\u0026rsquo;. (FS2020) (singleview)\nHmmm, lets look in the F# compiler source, were going to have to break out the big guns for this one.\nWhat we need to do is look at the different targets that are available. I remember seeing different targets when I was nosing through the source files a while ago. Also if you look at the readme.md file that comes with the open source compiler:\nYou can also build FSharp.Core for: .NET 2.0, Mono 2.1, Silverlight 5.0, Portable Profile47 (net4+sl4+wp71+win8) and XNA 4.0 for Xbox 360 profiles:\nmsbuild fsharp-library-build.proj /p:TargetFramework=net20 msbuild fsharp-library-build.proj /p:TargetFramework=mono21 msbuild fsharp-library-build.proj /p:TargetFramework=portable-net4+sl4+wp71+win8 msbuild fsharp-library-build.proj /p:TargetFramework=sl5 msbuild fsharp-library-build.proj /p:TargetFramework=net40-xna40-xbox360 So lets build the mono21 target with xbuild:\nxbuild fsharp-library-build.proj /p:TargetFramework=mono21 Now that\u0026rsquo;s build lets reference the output and see what happens:\nArrgh another error this time relating the the version of the framework that we have compiled against.\nIf you read the documentation for MonoTouch in a little more detail you will discover that a different mscorlib is required. We need to modify this in the build script:\nOpen up FSharp.Source.Targets and find the \u0026lt;PropertyGroup Condition=\u0026quot;'$(TargetFramework)'=='mono21'\u0026quot;\u0026gt; section, add the following after the \u0026lt;DefineConstants\u0026gt; elements.\n\u0026lt;OtherFlags\u0026gt;$(OtherFlags) --simpleresolution -r:\u0026quot;/Developer/MonoTouch/usr/lib/mono/2.1/mscorlib-runtime.dll\u0026quot; \u0026lt;/OtherFlags\u0026gt; Right, fingers crossed\u0026hellip;\nSigh, another error:\nError MT2002: Can not resolve reference: System.Reflection.Emit.AssemblyBuilder (MT2002) (singleview)\nWere getting closer though.\nLets look at the \u0026lt;DefineConstants/\u0026gt; that are declared in the build file, if you have a quick look you will notice that there is one called FX_NO_REFLECTION_EMIT that\u0026rsquo;s what we need so that Reflection.Emit is not included. MonoTouch does not support Reflection.Emit due to the meta data not being available once the code have been compiled with the AOT compiler.\nLets add that constant to the end of the rest:\n\u0026lt;DefineConstants\u0026gt;$(DefineConstants);FX_NO_REFLECTION_EMIT\u0026lt;/DefineConstants\u0026gt; If we rebuild Fsharp.Core again with xbuild and rebind the reference in our test project\u0026hellip;\nWow it works! # You should now have a working hello world application that can be deployed and run on real hardware.\nFinal Words # As this is just a documented hackathon I have mainly brain dumped what I remembered doing after the fact, so some steps may be slightly different. As soon as time permits Ill be adding a couple of project templates to the FSharpBinding to allow building F# MonoTouch libraries and applications.\nI also have some ideas for dealing with the UI and tooling with Xcode but Ill need a little time to investigate to see if it\u0026rsquo;s a viable option\u0026hellip;\nUntil next time!\n","date":"February 7, 2013","externalUrl":null,"permalink":"/programming/2013-02-04-monotouch-and-fsharp-part-ii/","section":"Blog","summary":"In the last post we left at the point where everything was running fine and dandy on the Simulator. So what happens if we compile for the real hardware?\nLets change the active configuration to Debug|iPhone and hit build, what do we get?\n","title":"MonoTouch and F# part II","type":"programming"},{"content":"MonoTouch and F# that would be a cool duo right?\nWell let me explain what needs to be done and why to get this pair working together.\nI heard rumours a while ago that F# and MonoTouch would not play together nicely because of limitations in the ahead of time compilation (AOT). So I thought I would either prove or disprove this with some concentrated hacking. How hard can it be?\nAs my good friend and colleague Dr. Kewin would quote:\n“No problem can withstand the assault of sustained thinking.”—Voltaire\nPrerequisites # These are the same as MonoTouch, I\u0026rsquo;m using a Mac and MonoDevelop at the moment. You would need a Mac anyway to be able to do the compile and deploy to an iOS device. Xcode with an Apple profile and certificates are required for code signing etc.\nFirst steps # So how do we tackle this?\nFirst lets look at the C# Single View MonoTouch project file (.csproj) up to the end of the first PropertyGroup:\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;utf-8\u0026#34;?\u0026gt; \u0026lt;Project DefaultTargets=\u0026#34;Build\u0026#34; ToolsVersion=\u0026#34;4.0\u0026#34; xmlns=\u0026#34;http://schemas.microsoft.com/developer/msbuild/2003\u0026#34;\u0026gt; \u0026lt;PropertyGroup\u0026gt; \u0026lt;Configuration Condition=\u0026#34; \u0026#39;$(Configuration)\u0026#39; == \u0026#39;\u0026#39; \u0026#34;\u0026gt;Debug\u0026lt;/Configuration\u0026gt; \u0026lt;Platform Condition=\u0026#34; \u0026#39;$(Platform)\u0026#39; == \u0026#39;\u0026#39; \u0026#34;\u0026gt;iPhoneSimulator\u0026lt;/Platform\u0026gt; \u0026lt;ProductVersion\u0026gt;10.0.0\u0026lt;/ProductVersion\u0026gt; \u0026lt;SchemaVersion\u0026gt;2.0\u0026lt;/SchemaVersion\u0026gt; \u0026lt;ProjectGuid\u0026gt;{822346B5-6805-42FD-9B6A-65446A688E63}\u0026lt;/ProjectGuid\u0026gt; \u0026lt;ProjectTypeGuids\u0026gt;{6BC8ED88-2882-458C-8E55-DFD12B67127B}; {FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}\u0026lt;/ProjectTypeGuids\u0026gt; \u0026lt;OutputType\u0026gt;Exe\u0026lt;/OutputType\u0026gt; \u0026lt;RootNamespace\u0026gt;HelloWorld\u0026lt;/RootNamespace\u0026gt; \u0026lt;AssemblyName\u0026gt;HelloWorld\u0026lt;/AssemblyName\u0026gt; \u0026lt;/PropertyGroup\u0026gt; The bits we are interested in are the ProjectTypeGuids. Visual Studio/MonoDevelop projects use these guid\u0026rsquo;s to identify the type of the project. If you do a bit of Googling (or Binging\u0026hellip;) you would find that:\n6BC8ED88-2882-458C-8E55-DFD12B67127B is a MonoTouch project type guid FAE04EC0-301F-11D3-BF4B-00C04F79EFBC is a C# project type guid The F# project type guid is F2A71F9B-5D33-465A-A702-920D77279786. We can now replace FAE04EC0-301F-11D3-BF4B-00C04F79EFBC with the F# one. For a comprehensive list of project type guid\u0026rsquo;s have a look at Mikhail Pilin\u0026rsquo;s blog. Next scroll down to the bottom of the project file and update the to . the final step on the project file is to change the project file extension from .csproj to .fsproj.\nThe last of the tweaking is to open up the .sln file and make a slight change to that too:\nMicrosoft Visual Studio Solution File, Format Version 11.00 # Visual Studio 2010 Project(\u0026quot;{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}\u0026quot;) = \u0026quot;singleview\u0026quot;, \u0026quot;singleview\\singleview.fsproj\u0026quot;, \u0026quot;{4465399C-4EE8-4F60-AD9A-EB9AEDD1C5BF}\u0026quot; EndProject Global ...snip... Modify the Project sections Guid FAE04EC0-301F-11D3-BF4B-00C04F79EFBC to the F# project type Guid 4925A630-B079-445d-BCD4-3A9C94FE9307. If you forget this step then MonoDevelop will get really confused and try to compile the F# project with the C# compiler.\nCode Changes # For the sake of simplicity I\u0026rsquo;m going to port the C# code verbatim showing the C# code first then the F# code. The easiest way would probably be to change all the C# files to have the .fs extension and edit them in place, remembering to also update the entries in the .fsproj file too this only takes a second to do.\nI know what you are going to say: \u0026ldquo;Why didn\u0026rsquo;t you just create a nice project template for us all to use?\u0026rdquo;\nI am, I am, patience!\nA number of people wanted to know what I did to get things going so this is my documented \u0026lsquo;hack-a-thon\u0026rsquo; if you like. The project template will be along shortly. Lets move along to the code changes.\nViewController # using System; using System.Drawing; using MonoTouch.Foundation; using MonoTouch.UIKit; namespace singleview { public partial class singleviewViewController : UIViewController { public singleviewViewController () : base (\u0026#34;singleviewViewController\u0026#34;, null) { } public override void DidReceiveMemoryWarning () { // Releases the view if it doesn\u0026#39;t have a superview. base.DidReceiveMemoryWarning (); // Release any cached data, images, etc that aren\u0026#39;t in use. } public override void ViewDidLoad () { base.ViewDidLoad (); // Perform any additional setup after loading the view, typically from a nib. } public override void ViewDidUnload () { base.ViewDidUnload (); // Clear any references to subviews of the main view in order to // allow the Garbage Collector to collect them sooner. // e.g. myOutlet.Dispose (); myOutlet = null; ReleaseDesignerOutlets (); } public override bool ShouldAutorotateToInterfaceOrientation (UIInterfaceOrientation toInterfaceOrientation) { // Return true for supported orientations return (toInterfaceOrientation != UIInterfaceOrientation.PortraitUpsideDown); } } } // This file has been generated automatically by MonoDevelop to store outlets and // actions made in the Xcode designer. If it is removed, they will be lost. // Manual changes to this file may not be handled correctly. using MonoTouch.Foundation; namespace singleview { [Register (\u0026#34;singleviewViewController\u0026#34;)] partial class singleviewViewController { void ReleaseDesignerOutlets () { } } } namespace Singleview open System open System.Drawing open MonoTouch.Foundation open MonoTouch.UIKit [\u0026lt;Register (\u0026#34;singleviewViewController\u0026#34;)\u0026gt;] type singleviewViewController() = inherit UIViewController(\u0026#34;singleviewViewController\u0026#34;, null) let ReleaseDesignerOutlets() = ( (* No outlets to release *)) override x.DidReceiveMemoryWarning() = // Releases the view if it doesn\u0026#39;t have a superview. base.DidReceiveMemoryWarning(); // Release any cached data, images, etc that aren\u0026#39;t in use. override x.ViewDidLoad() = base.ViewDidLoad() // Perform any additional setup after loading the view, typically from a nib. override x.ViewDidUnload() = base.ViewDidUnload() // Clear any references to subviews of the main view in order to // allow the Garbage Collector to collect them sooner. // e.g. myOutlet.Dispose (); myOutlet = null; ReleaseDesignerOutlets() override x.ShouldAutorotateToInterfaceOrientation(toInterfaceOrientation) = // Return true for supported orientations toInterfaceOrientation \u0026lt;\u0026gt; UIInterfaceOrientation.PortraitUpsideDown On looking at this section you will notice that there is no partial class in the F# version, that\u0026rsquo;s because F# doesn\u0026rsquo;t have any notion of partial classes. In this simple project we don\u0026rsquo;t actually have any interaction with the UI so designer interaction is a moot point at the moment.\nThe .fsproj file still needs to be edited to remove the nested partial class that is present in the C# version:\n\u0026lt;Compile Include=\u0026#34;singleviewViewController.designer.cs\u0026#34;\u0026gt; \u0026lt;DependentUpon\u0026gt;singleviewViewController.cs\u0026lt;/DependentUpon\u0026gt; \u0026lt;/Compile\u0026gt; Simply remove the DependUpon element and just use the name singleviewViewController.fs:\n\u0026lt;Compile\u0026gt;singleviewViewController.fs/\u0026gt; The lack of partial classes in F# makes the tooling available for UI designer a pain to integrate tightly into F# without a bit of work work (I have some ideas on that that I\u0026rsquo;m currently experimenting with that Ill return to after finishing this article). Currently MonoTouch uses the Xcodes interface designer to build the UI which is stored in a xib file. This is simply a file describing the user interface and its interaction points. The Properties of the UI are called Outlets and events spawned from the UI are called Actions.\nAppDelegate # using System; using System.Collections.Generic; using System.Linq; using MonoTouch.Foundation; using MonoTouch.UIKit; namespace singleview { // The UIApplicationDelegate for the application. This class is responsible for launching the // User Interface of the application, as well as listening (and optionally responding) to // application events from iOS. [Register (\u0026#34;AppDelegate\u0026#34;)] public partial class AppDelegate : UIApplicationDelegate { // class-level declarations UIWindow window; singleviewViewController viewController; // This method is invoked when the application has loaded and is ready to run. In this // method you should instantiate the window, load the UI into it and then make the window visible. // You have 17 seconds to return from this method, or iOS will terminate your application. public override bool FinishedLaunching (UIApplication app, NSDictionary options) { window = new UIWindow (UIScreen.MainScreen.Bounds); viewController = new singleviewViewController (); window.RootViewController = viewController; window.MakeKeyAndVisible (); return true; } } } namespace Singleview open System open System.Collections.Generic open MonoTouch.Foundation open MonoTouch.UIKit // The UIApplicationDelegate for the application. This class is responsible for launching the // User Interface of the application, as well as listening (and optionally responding) to application events from iOS. [\u0026lt;Register (\u0026#34;AppDelegate\u0026#34;)\u0026gt;] type AppDelegate() = inherit UIApplicationDelegate() let mutable window = Unchecked.defaultof\u0026lt;_\u0026gt; let mutable viewController = Unchecked.defaultof\u0026lt;_\u0026gt; // This method is invoked when the application has loaded and is ready to run. In this // method you should instantiate the window, load the UI into it and then make the window visible. // You have 17 seconds to return from this method, or iOS will terminate your application. override x.FinishedLaunching ( app: UIApplication, options: NSDictionary) = window \u0026lt;- new UIWindow(UIScreen.MainScreen.Bounds) viewController \u0026lt;- new singleviewViewController() window.RootViewController \u0026lt;- viewController window.MakeKeyAndVisible() true The code is pretty similar between the two implementations, with the F# version omitting the type annotations, semicolons and curly braces. The other area to notice is that the mutable variable declarations for the window and viewController bindings. The C# implementation defaults to mutable variables whereas F# defaults to the safer immutable ones.\nProgram/main # using System; using System.Collections.Generic; using System.Linq; using MonoTouch.Foundation; using MonoTouch.UIKit; namespace singleview { public class Application { // This is the main entry point of the application. static void Main (string[] args) { // if you want to use a different Application Delegate class from \u0026#34;AppDelegate\u0026#34; // you can specify it here. UIApplication.Main (args, null, \u0026#34;AppDelegate\u0026#34;); } } } module main open System open System.Collections.Generic open MonoTouch.Foundation open MonoTouch.UIKit [\u0026lt;EntryPoint\u0026gt;] let main( args) = UIApplication.Main (args, null, \u0026#34;AppDelegate\u0026#34;) 0 The main thing you will notice is that the F# code is terser, again dropping the type annotations, semicolons and curly braces. Oh, I also called the entry point main. To be precise it\u0026rsquo;s a function called main in a module named main, there\u0026rsquo;s no need to create a class or type for this.\nThe Xib file # In C# MonoToch projects the xib file is compiled and embedded for you as part of the build process, unfortunately this is not currently possible in F# so we have to do it manually. In an ideal world this would all be done by the F# project at build time and this is something that I\u0026rsquo;m working on too. In the mean time we have to do it manually so open up your trusty friend the Terminal.\nI\u0026rsquo;m going to split the command line into separate parts due to its size:\nFirst of all we invoke the ibtool:\n/Applications/Xcode.app/Contents/Developer/usr/bin/ibtool --errors --warnings --notices --output-format human-readable-text --compile Followed by name of the .nib file you want to compile to:\n\u0026quot;/yourPath/singleviewViewController.nib\u0026quot; The path of the .xib you want to compile from:\n\u0026quot;/yourPath/singleviewViewController.xib\u0026quot; Finally the sdk that you want to use for compilation, in this instance it is The iPhoneSimulator6.0.sdk as we are targetting the simulator: \u0026ndash;sdk \u0026ldquo;/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator6.0.sdk\u0026rdquo;\nOnce you have compiled to a .nib file include it in the project, set the build action to Content. You can still include the .xib version within the project if you want but you would have to set the build action to None. Currently the F# binding does not support the build action of Interface Definition if it did then we probably wouldn\u0026rsquo;t have to go through the manual compilation process either.\nThat ought to do it, everything should now work on the simulator. If you try to compile to a real phone then everything will quickly come grinding to a halt but Ill explain all of that next time and how to resolve it too.\nUntil next time!\n","date":"February 3, 2013","externalUrl":null,"permalink":"/programming/2013-02-03-monotouch-and-fsharp-part-i/","section":"Blog","summary":"MonoTouch and F# that would be a cool duo right?\nWell let me explain what needs to be done and why to get this pair working together.\nI heard rumours a while ago that F# and MonoTouch would not play together nicely because of limitations in the ahead of time compilation (AOT). So I thought I would either prove or disprove this with some concentrated hacking. How hard can it be?\n","title":"MonoTouch and F# part I","type":"programming"},{"content":"","date":"January 4, 2013","externalUrl":null,"permalink":"/tags/mac/","section":"Tags","summary":"","title":"Mac","type":"tags"},{"content":"","date":"January 4, 2013","externalUrl":null,"permalink":"/tags/monogame/","section":"Tags","summary":"","title":"Monogame","type":"tags"},{"content":"I\u0026rsquo;ve been on a bit of a break from my normal jovial self due to a shit storm of bad stuff happening that I wont go into here, but hopefully this years going to be awesome. Anyway, here\u0026rsquo;s the next exciting installment in my series on MonoGame. (Well I find it exciting anyway :-) )\nIf you remember back to the last post I mentioned the platonic solids, and we created and rendered the tetrahedron, lets recap on what the five solids are:\nTetrahedron (four faces) Cube or hexahedron (six faces) Octahedron (eight faces) Dodecahedron (twelve faces) Icosahedron (twenty faces) We covered the tetrahedron in the previous post and the hexahedron is pretty humdrum so I\u0026rsquo;m not going to cover that here so lets move onto the next one the octahedron.\nCreating the Octahedron # Here\u0026rsquo;s a function that we will use to generate an octahedron:\nmodule Platonic let createOctahedron()= let top = Vector3.Up let midOne = top |\u0026gt; Vector3.transform (Matrix.CreateRotationX(toRad 90.0f) * Matrix.CreateRotationY(toRad 45.f)) let midTwo = top |\u0026gt; Vector3.transform (Matrix.CreateRotationX(toRad 90.0f) * Matrix.CreateRotationY(toRad 135.f)) let midThree = top |\u0026gt; Vector3.transform (Matrix.CreateRotationX(toRad 90.0f) * Matrix.CreateRotationY(toRad 225.f)) let midFour = top |\u0026gt; Vector3.transform (Matrix.CreateRotationX(toRad 90.0f) * Matrix.CreateRotationY(toRad 315.f)) let bottom = top |\u0026gt; Vector3.transform (Matrix.CreateRotationX(toRad 180.f)) [| midOne; top; midTwo midTwo; top; midThree midThree; top; midFour midFour; top; midOne midOne; midTwo; bottom midTwo; midThree; bottom midThree; midFour; bottom midFour; midOne; bottom |] You can see that the bulk of the code is centred around rotating a Y axis unit vector top around the X and Y axis. All the vertices around the centre od the octahedron lie on the same plain and are simply rotated by 90 degrees in the X axis and then rotated by multiples of 90 degrees in the Y axis starting at 45 degrees (45, 135, 225, 315). Finally the the top unit vector is flipped to the bottom by rotating around 180 degrees in the X axis, this forms the bottom point. The final step consists of combining the vertices into an array with the array syntax [| ... |] specifing each triangle of the octahedron in turn.\nIf you were looking carefully you might have noticed that the Vector3.transform function is not part of the MonoGame library. I wrapped MonoGames\u0026rsquo;s Vector3.Transform function so that the Vector3 is the last parameter so we can use the forward pipeline operator |\u0026gt;:\nmodule Vector3 = let transform (m:Matrix) v = Vector3.Transform(v, m) Drawing the Octahedron # What now? Well, with this code we have just been working with the raw vertices, we now need to get this into a form that MonoGame can render, namely an array of the VertexPositionColor structure. It\u0026rsquo;s a bit of a mouthful so lets alias this so we can simply refer to it as vpc:\nlet vpc v c = VertexPositionColor(v, c) To render the octahedron we can now modify the draw method of the tetrahedron code from the last post maybe something like this should illustrate:\noverride x.Draw (gameTime) = // Clear the backbuffer x.GraphicsDevice.Clear (Color.CornflowerBlue) for pass in basicEffect.CurrentTechnique.Passes do pass.Apply() let octahedron = Platonic.createOctahedron() |\u0026gt; Array.mapi (fun i -\u0026gt; Platonic.vpc (if i % 2 = 0 then Color.BlueViolet else Color.Orange) ) x.GraphicsDevice.DrawUserPrimitives(PrimitiveType.TriangleList, octahedron, 0, octahedron.Length / 3) Bear in mind we are not looking at optimisation at all at this stage purely visualising what we have. We are using the mapi function to alternate between defining blue violet and orange vertex colours. At the moment because we haven\u0026rsquo;t set up any lights the octahedron would just appear as diamond chunk of colour with no shading, with these two simple vertex colours we can see the separate facets and see the 3D form.\nSubdivision Surfaces # According to Wikipedia:\nA subdivision surface, in the field of 3D computer graphics, is a method of representing a smooth surface via the specification of a coarser piecewise linear polygon mesh. The smooth surface can be calculated from the coarse mesh as the limit of a recursive process of subdividing each polygonal face into smaller faces that better approximate the smooth surface.\nThey are also known as scalable geometry. I\u0026rsquo;m not going to get into the realm of true sub-divisional modelling such as providing a visible control surface with editing and crease support, I just wouldn\u0026rsquo;t be able to do it justice within the scope of this introductory series. Applications like Softimage or Maya are masters of sub-divisional modelling, you might want to check those out if you are interested in what can be done in that area. Subdivision surfaces have been quite popular in the computer graphics industry as it allows modellers and animators to work with simple mesh surfaces with far less control points that can be rendered with super smooth detail but without the constraints of having to work with millions of points on the screen at once which can be computationally very expensive and distracting. Nowadays that kind of processing is done by a GPU\u0026rsquo;s vertex shader\u0026rsquo;s or more recently the geometry shader\u0026rsquo;s which can take a simple triangle as an input and produce zero or more triangles as its output.\nOne of the properties of platonic solids is that all of the defining vertices lie on a sphere. If we were to take each of the defining faces or triangles and recursively divide them into four smaller triangles, and project each of the containing vertices onto the sphere then eventually we would get an approximation of a sphere. This was the basis of Charles Loop\u0026rsquo;s thesis Smooth Subdivision Surfaces Based on Triangles. What I am going to present here will not go into that level of detail and we will not be generating any control surfaces to act on the subdivision mesh. We could call this a poor man\u0026rsquo;s subdivision surface or sphere approximation :-).\nLets create a quick and dirty function to try this out anyway:\nlet rec subdivide(v1, v2, v3, depth) = seq{match depth with | 0 -\u0026gt; yield vpc Color.LightBlue (v1 |\u0026gt; Vector3.Normalize) yield vpc Color.AliceBlue (v2 |\u0026gt; Vector3.Normalize) yield vpc Color.SlateGray (v3 |\u0026gt; Vector3.Normalize) | _ -\u0026gt; let u12 = ((v1 + v2) / 2.0f) |\u0026gt; Vector3.Normalize let u23 = ((v2 + v3) / 2.0f) |\u0026gt; Vector3.Normalize let u31 = ((v3 + v1) / 2.0f) |\u0026gt; Vector3.Normalize yield! subdivide(v1, u12, u31, depth-1) yield! subdivide(v2, u23, u12, depth-1) yield! subdivide(v3, u31, u23, depth-1) yield! subdivide(u12, u23, u31, depth-1) } Here we have a recursive function that takes three vertices v1, v2, v3 and a depth parameter. When the depth parameter is zero we are at our subdivision maximum and we return a normalized triangle. Incidentally for the same lighting issues mentioned above we use three different colours for the vertices: light blue, alice blue, and slate grey. The three vertices u12, u23, u31 define the points in-between the input triangle, we calculate them by adding the vertices together and dividing them by two ((v1 + v2) / 2.0f) then pipe-lining the result to the normalize function (|\u0026gt; Vector3.Normalize). We do this for each of the points. The final step is the yield! section which creates the next level of subdivision for each of the resulting four triangles. Remember our input triangle is divided into four. If fact in the previous article there are several images of this:\nTetrahedron-coordinates\nTetrahedron\nThe Sierpinski triangle (without the holes) is actually our subdivision method, except the we subdivide every triangle produced.\nTo try this out lets change the Draw method so that it looks like this:\noverride x.Draw (gameTime) = x.GraphicsDevice.Clear (Color.CornflowerBlue) for pass in basicEffect.CurrentTechnique.Passes do pass.Apply() let subdiv = Platonic.createOctahedron() |\u0026gt; Seq.windowed 3 |\u0026gt; Seq.map (function | [|a;b;c|] -\u0026gt; subdivide(a,b,c, 3) | _ -\u0026gt; failwith \u0026#34;Unsupported array size.\u0026#34; ) |\u0026gt; Seq.concat |\u0026gt; Seq.toArray x.GraphicsDevice.DrawUserPrimitives(PrimitiveType.TriangleList, subdiv, 0, subdiv.Length / 3) Here we are using some of the functions from the sequence module to group and process the vertices.\nFirst the result of Platonic.createOctahedron() is grouped into triangles using Seq.windowed 3. Now we map each the triangle using the using the subdivide function. Next we merge the sequence back together using Seq.concat. Finally we convert the sequence back into an array with Seq.toArray. The image below shows the octahedron at various levels of subdivision from one through to four: Well I hope you enjoyed this brief sojourn into subdivision, if you want to investigate further I recommend looking at the following papers.\nRecursively generated B-spline surfaces on arbitrary topological meshes\nSmooth Subdivision Surfaces Based on Triangles\nEvaluation of Loop Subdivision Surfaces\nIt\u0026rsquo;s a very interesting area and I dont think will be able to resist doing another article delving deaper later on.\nUntil next time\u0026hellip;\n","date":"January 4, 2013","externalUrl":null,"permalink":"/programming/2013-01-04-monogame-subdivision-and-platonics/","section":"Blog","summary":"I’ve been on a bit of a break from my normal jovial self due to a shit storm of bad stuff happening that I wont go into here, but hopefully this years going to be awesome. Anyway, here’s the next exciting installment in my series on MonoGame. (Well I find it exciting anyway :-) )\n","title":"MonoGame subdivision and platonics","type":"programming"},{"content":"This time we are going to get into a little bit of code and produce the simplest of all 3d solids, the tetrahedron. I know its not the most exciting of things but we have to start somewhere. The scope of 3D graphics in computers is so vast that its very easy to get lost in the vast piles of research papers.\nFirst lets do some basic setup, if you followed my last post then you will will have a project template to use, this makes this a little easier.\nFor those of you that are running on Windows and want to use Visual Studio please leave a comment if you would like a project template for F#. The beauty of MonoGame is that it is cross platform and there is only a small amount of code that differers between the different platforms, and that is localised to the main entry point rather than the Game type.\nFirst create a new MonoGame Mac Application project, you should end up with a Game1 type that looks like this:\ntype MonoGame3DBasics() as x = inherit Game() let graphics = new GraphicsDeviceManager(x) let mutable spriteBatch = Unchecked.defaultof\u0026lt;_\u0026gt; let mutable logoTexture = Unchecked.defaultof\u0026lt;_\u0026gt; do x.Content.RootDirectory \u0026lt;- \u0026#34;Content\u0026#34; graphics.IsFullScreen \u0026lt;- false /// Overridden from the base Game.Initialize. Once the GraphicsDevice is setup, /// we\u0026#39;ll use the viewport to initialize some values. override x.Initialize() = base.Initialize() /// Load your graphics content. override x.LoadContent() = // Create a new SpriteBatch, which can be use to draw textures. spriteBatch \u0026lt;- new SpriteBatch (graphics.GraphicsDevice) // TODO: use this.Content to load your game content here eg. logoTexture \u0026lt;- x.Content.Load\u0026lt;_\u0026gt;(\u0026#34;logo\u0026#34;) /// Allows the game to run logic such as updating the world, /// checking for collisions, gathering input, and playing audio. override x.Update ( gameTime:GameTime) = // TODO: Add your update logic here base.Update (gameTime) /// This is called when the game should draw itself. override x.Draw (gameTime:GameTime) = // Clear the backbuffer graphics.GraphicsDevice.Clear (Color.CornflowerBlue) spriteBatch.Begin() // draw the logo spriteBatch.Draw (logoTexture, Vector2 (130.f, 200.f), Color.White); spriteBatch.End() //TODO: Add your drawing code here base.Draw (gameTime) We are going to need a few extra field for this sample and we wont be using and 2d so remove the spriteBatch and the logoTexture as we wont be needing those. The following fields need to be added in their place:\nlet mutable basicEffect = Unchecked.defaultof\u0026lt;_\u0026gt; let mutable texture = Unchecked.defaultof\u0026lt;_\u0026gt; let mutable vertexBuffer = Unchecked.defaultof\u0026lt;_\u0026gt; let mutable view = Unchecked.defaultof\u0026lt;_\u0026gt; let mutable world = Unchecked.defaultof\u0026lt;_\u0026gt; let mutable projection = Unchecked.defaultof\u0026lt;_\u0026gt; What\u0026rsquo;s this! mutable fields! I know, but this simplifies things until I can put together a friendly functional scaffolding around MonoGame. We are creating a basicEffect, this is used to draw the 3D objects, its actually just a basic shader implementation with simple lighting. We also have texture which will be used as our texture map. We have a vertexBuffer which is used to store the vertices for out primitive. view, world, and projection are our matrices which are used to look into our 3D scene. For more information on the theory behind 3D projection have a look here.\nLets more to the LoadContent override:\noverride x.LoadContent() = //load texture texture \u0026lt;- x.Content.Load\u0026lt;Texture2D\u0026gt;(\u0026#34;Tetrahedron\u0026#34;) //world, view, projection world \u0026lt;- Matrix.Identity view \u0026lt;- Matrix.CreateLookAt(Vector3(0.f, 0.f, 10.f), Vector3.Zero, Vector3.Up) projection \u0026lt;- Matrix.CreatePerspectiveFieldOfView(MathHelper.PiOver4, x.GraphicsDevice.Viewport.AspectRatio, 1.f, 1000.f) basicEffect \u0026lt;- new BasicEffect(x.GraphicsDevice, World = world, View = view, Projection = projection, Texture = texture, TextureEnabled = true) let tetrahedronData = generateTetrahedron 3.5f vertexBuffer \u0026lt;- new VertexBuffer(x.GraphicsDevice, VertexPositionTexture.VertexDeclaration, tetrahedronData.Length, BufferUsage.WriteOnly) vertexBuffer.SetData(tetrahedronData) x.GraphicsDevice.SetVertexBuffer(vertexBuffer) The first thing that we do is load the texture map:\ntexture \u0026lt;- x.Content.Load\u0026lt;Texture2D\u0026gt;(\u0026quot;Tetrahedron\u0026quot;)\nThis simply loads in the texture named Tetrahedron using the content loader.\nNext we set up the default values for the world matrix, view and projection matrices. The world is simply initialised using Matrix.Identity which is a matrix defined as:\n[1,0,0,0] [0,1,0,0] [0,0,1,0] [0,0,0,1] The view is initialised using the CreateLookAt method of the Matrix type. This sets up a transformation that points from 0,0,10 to the centre of the world using Vector3.Zero. It also uses the Vector3.Up as the orientation direction (Positive Y is up).\nThe projection is also initialised using the helper method CreatePerspectiveFieldOfView which as you might guess, creates a perspective with a field of view. In this instance our field of view uses the constant PiOver4.\nThe basic effect is now initialised with the matrices we just initialised.\nFor now I want you to ignore the line let tetrahedronData = generateTetrahedron 3.5f. I need to explain how to generate a tetrahedron before that will make sense, just assume that is returns the vertices that we need for the tetrahedron.\nThe vertexBuffer is now created which will hold all the vertices for the tetrahedron. We need to tell the vertexBuufer what format we want to use to hold the vertices, here, we are going to use vertices with Position, Colour, and Texture coordinates so we we use the predefined format of VertexPositionTexture.VertexDeclaration. There are various different predefined formats and its also possible to create custom user defined formats, for more information have a look here. I realise I\u0026rsquo;m glossing over a lot of information, this is because the field of 3D graphics is huge even an API such as XNA/MonoGame which tries to simplify things, there is still a vast array of different concepts and I don\u0026rsquo;t want to get too bogged down with all the specifics.\nFinally the vertexBuffer is assigned to the graphics device: x.GraphicsDevice.SetVertexBuffer(vertexBuffer), this loads the vertex buffer into the graphics card ready to be draw later.\nNext we move on to the Update override:\noverride x.Update(gameTime) = if Keyboard.GetState().IsKeyDown(Keys.Escape) then x.Exit() let time = float32 gameTime.ElapsedGameTime.TotalSeconds // Compute camera matrices. let rotationz = Matrix.CreateRotationY(time * 1.2f) basicEffect.View \u0026lt;- rotationz * Matrix.CreateLookAt(Vector3(0.f, 0.f, 10.f), Vector3.Zero, Vector3.Up) base.Update (gameTime) The Update method is called every time the game decides that game logic needs to be processed. This includes the management of game state, the processing of user input, and also the updating of simulation data or AI.\nFirst of all we check the Escape key has been pressed so that the application can exit: if Keyboard.GetState().IsKeyDown(Keys.Escape) then x.Exit().\nNext we capture the amount of elapsed time since the last update so that we can calculate distance moved over time etc.\nTo make our view of the world less static we create a rotation around the z axis of the world so that we see the tetrahedron from different angles. We multiply the rotation matrix by our initial Matrix.CreateLookAt... that we used earlier on, and assign it back to the View property of the basicEffect. I want to stress that the aim of this is not super optimal code it\u0026rsquo;s merely to show the easiest possible method of achieving a result. In a future post we will be looking at some functional scaffolding to allow us to apply functional thinking to this domain. Perhaps introducing a small Domain Specific Languagee to help.\nFinally we have the Draw override:\n/// This is called when the game should draw itself. override x.Draw (gameTime) = // Clear the backbuffer x.GraphicsDevice.Clear (Color.CornflowerBlue) for pass in basicEffect.CurrentTechnique.Passes do pass.Apply() x.GraphicsDevice.DrawPrimitives(PrimitiveType.TriangleList, 0, 4) base.Draw (gameTime) The Draw override is called every time the game needs to draw a frame, we put all out rendering code in here.\nThe first step is to clear the screen to a nice blue colour:\nx.GraphicsDevice.Clear (Color.CornflowerBlue)\nTo draw our tetrahedron we need to loop through the different techniques in out shader (In this instance our basicEffect only has 1), apply the technique, then draw out triangles. You might remember earlier to created a the vertexBuffer and assigned it to the graphics device. All we have to do is tell MonoGame that we want to draw 4 triangles and they are in a TriangleList.\nThat\u0026rsquo;s it all done! Well almost, now lets backtrack slightly and look at how we build the vertices for that tetrahedron.\nBuilding a tetrahedron # What is a tetrahedron? Well if you look on wikipedia\nA tetrahedron is a polyhedron composed of four triangular faces, three of which meet at each vertex. It has six edges and four vertices. The tetrahedron is the only convex polyhedron that has four faces. \u0026hellip;\nIn the case of a tetrahedron the base is a triangle(any of the four faces can be considered the base), so a tetrahedron is also known as a \u0026ldquo;triangular pyramid\u0026rdquo;. \u0026hellip;\nFor any tetrahedron there exists a sphere (the circumsphere) such that the tetrahedron\u0026rsquo;s vertices lie on the sphere\u0026rsquo;s surface.\nThe tetrahedron is also the simplest of the five platonic solids. There are lots of interesting properties of these but I don\u0026rsquo;t really want to go into that here we just want to draw and texture one for now.\nSo how do we construct a tetrahedron?\nThere are various methods that can be used to construct a tetrahedron ranging from formula such as:\nCartesian coordinate based (±1, 0, -1/sqrt2) (0, ±1, 1/sqrt2)\nV0 =(0,0,1) V1=(2sqrt2 /3, 0, −1/3)\nV2 =(− sqrt2 /3, sqrt6 /3, −1/3)\nV3=(− sqrt2 /3,− sqrt6 /3,−1/3)\nYes I know I need to get latex maths expression working in my blog! Ill have to work on that.\nI don\u0026rsquo;t know about you, but I always feel uneasy unless I can clearly see exactly what\u0026rsquo;s been done, I also don\u0026rsquo;t want to turn this into a 3D geometry lesson because that\u0026rsquo;s not what I intend this post to be about.\nHere\u0026rsquo;s what works for me anyway.\nCalculate the radius of the circumsphere, this is the sphere in which all of the vertices of the tetrahedron sit, this is calculated by sqrt 3/8.\nThe angle between each vertex and its centre point is acos -1/3 or ~ 109.471 degrees.\nThe first vertex is (0, sqrt (3/8) * length, 0) To get our second vertex we need to rotate the first vertex by acos -1/3 in the X axis To get the third vertex we rotate second vertex by 120 degrees in the Y axis For the last vertex we rotate the second vertex by -120 degrees in the Y axis A picture can often be a worth a thousand words, I think this is one of those times, I will refer you to Friedrich A. Lohmüllers site for an excellent pictorial and description. The code for this process is below.\nlet generateTetrahedron size = let circumSphereRadius = sqrt (3.f/8.f) * size let centerVertexAngle = acos (-1.f/3.f) let v1 = Vector3(0.f, circumSphereRadius, 0.f) let v2 = v1 |\u0026gt; Vector3.rotateX centerVertexAngle let v3 = v2 |\u0026gt; Vector3.rotateY (radians 120.f) let v4 = v2 |\u0026gt; Vector3.rotateY (-radians 120.f) let uv1 = Vector2(0.5f, 1.f - sqrt 0.75f) let uv2 = Vector2(0.75f, 1.f - (sqrt 0.75f)/2.f) let uv3 = Vector2(0.25f, 1.f - (sqrt 0.75f)/2.f) let uv4 = Vector2(0.5f, 1.f) let uv5 = Vector2.UnitY let uv6 = Vector2.One [| VertexPositionTexture(v1, uv1) VertexPositionTexture(v3, uv2) VertexPositionTexture(v2, uv3) VertexPositionTexture(v1, uv2) VertexPositionTexture(v4, uv6) VertexPositionTexture(v3, uv4) VertexPositionTexture(v1, uv3) VertexPositionTexture(v2, uv4) VertexPositionTexture(v4, uv5) VertexPositionTexture(v2, uv3) VertexPositionTexture(v3, uv2) VertexPositionTexture(v4, uv4) |] The last piece of the puzzle is the texture coordinates. There is some amazing software available to help model both texture and 3d geometry, projecting the vertices onto a 2d plane can be an art-form in itself. Luckily the tetrahedron is one of the simplest models, if you imagine the tetrahedron unfolded it would look like this from the top:\nTo map a texture to the tetrahedron we have to include a texture coordinate with every vertex. These coordinates are uv1-uv6 in the code above. We use some ratios to select the correct coordinates within the texture. The texture coordinates are always between 0 and 1. Here\u0026rsquo;s the location of the above points so you can see the locations clearly.\nTo make sure that the texture is in the right place we are going to use a type of fractal called the Sierpinski triangle. The Sierpinski triangle had exactly the same net, or unfolded shape as the texture we need to use. Each of the first iterations of the fractal is coloured separately as this will make it easy to see if the mapping is correct. Here is what the texture looks like:\nThis is how everything will look, I know its not incredibly impressive but its MonoGame in 3d, F#, and all running on a Mac, what more could you want! ;)\nIts feels like we covered a lot of ground here but all we have is a spinning tetrahedron, its tricky to know what level of detail to go down to. I don\u0026rsquo;t want to teach anyone how to suck eggs, and I want to alienate people who are new to this area and want to learn, I hope I got the balance about right.\nIf you want to just get the code and have a look then here\u0026rsquo;s my GitHub repo\nAs usual I appreciate any comments and feedback.\nUntil next time!\n","date":"November 25, 2012","externalUrl":null,"permalink":"/programming/2012-11-25-monogame-3d-basics/","section":"Blog","summary":"This time we are going to get into a little bit of code and produce the simplest of all 3d solids, the tetrahedron. I know its not the most exciting of things but we have to start somewhere. The scope of 3D graphics in computers is so vast that its very easy to get lost in the vast piles of research papers.\n","title":"MonoGame 3D basics","type":"programming"},{"content":"What we are going to do in this post is take a whistle stop tour of getting MonoGame up and running along with a simple demo in F#. Over the last few days I have been building an F# project template for MonoDevelop, this post will also how to get that installed too. First of all I\u0026rsquo;m going to assume that you have the following installed:\nMono 3.0 beta MonoDevelop MonoDevelop F# language binding If you don\u0026rsquo;t have a look at my previous post that explains all that, if you don\u0026rsquo;t want to build the F# binding from source then you can use the Add-in Manager. If you look in Gallery the language binding section contains the F# language binding. I prefer using the source at the moment as I like to tweak a few things here and there and nosy around in the code.\nCloning and building # First clone my MonoGame repo, why my repo? Well, I have been doing F# specific work and I have been submitting my pull requests but there will always be a lag while I\u0026rsquo;m waiting for one of the maintainers to merge in my code, an I might fix a bug in between writing this blog and you reading it. The main repo is located here if you are interested in looking at that too.\ngit clone git@github.com:7sharp9/MonoGame.git OK, now we need to initialise and get the submodules:\ngit submodule init git subbmodule update Now go and make a cup of tea or coffee or something because this will take a little while! :-)\nif you get any serious errors from the submodules you cold try this:\ngit submodule sync git submodule update Check that the main framework builds, this will test if all the submodules updated correctly as well.\ncd MonoGame xbuild MonoGame.Framework.MacOS.sln Building the Project templates # I was going to go through the manual steps required to get things up and running but I figured I may as well go through getting the project templates installed too. Now the project templates are not 100% by any means and there is currently work under way to produce a cross platform installer that will take care of everything.\nThere are currently the following C# based templates:\nMonoGame for Android MonoGame for iOS MonoGame for Windows Application MonoGame for Linux MonoGame for Mac Application And the following brand new shiny F# templates that I added a few days ago:\nMonoGame for Mac Application Yeah I know there\u0026rsquo;s only one for now but Ill get there\u0026hellip;\nRight, we\u0026rsquo;re going to rattle through some commands to build the project templates:\n#Change directory to the project templates cd ProjectTemplates/MonoDevelop/MonoDevelop.MonoGame/ #Build the Project templates solution: xbuild MonoDevelop.MonoGame.sln Now we\u0026rsquo;re going to create the add-in structure and install it in MonoDevelop\n#create a folder for the add-in: mkdir /Applications/MonoDevelop.app/Contents/MacOS/lib/monodevelop/AddIns/MonoDevelop.MonoGame #Copy the templates and icons into that folder: cp -R MonoDevelop.MonoGame/templates /Applications/MonoDevelop.app/Contents/MacOS/lib/monodevelop/AddIns/MonoDevelop.MonoGame cp -R MonoDevelop.MonoGame/icons /Applications/MonoDevelop.app/Contents/MacOS/lib/monodevelop/AddIns/MonoDevelop.MonoGame cp -R MonoDevelop.MonoGame/bin/Release/MonoDevelop.MonoGame.dll /Applications/MonoDevelop.app/Contents/MacOS/lib/monodevelop/AddIns/MonoDevelop.MonoGame If you fire up MonoDevelop and create a new solution now you should see a new project type:\nI know this is a bit long winded but it will get you started until a proper installer finished.\nOh yeah there are currently a couple of caveats, once you have created a new project from the template you must do the following:\nOpen the project references and re-add the Lidgren.Network.dll and the MonoGame.FrameWork.MacOS.dll references.\nI know I know that\u0026rsquo;s a major pain but let me tell you the reason for this.\nPackage references in MonoDevelop # You might of noticed that the aforementioned references were shown in red when the new project was opened, this was because there was no way for MonoDevelop to know where they were currently located. We could of added a package config file to the /Library/Frameworks/Mono.framework/Versions/3.0.0/lib/pkgconfig folder which would allow MonoDevelop to know how to resolve the references but this then leads to a further problem:\nPackage references in MonoDevelop are expected to be located in the GAC, and because of this assumption the copy local property has no effect. There is an oustanding bug logged for this issue. There are workarounds for these issues as mentioned in the bug report but it is still currently an issue in MonoGame. Hopefully it will be addressed in the near future.\nYou might be asking why cant we put the assemblies it in the GAC?\nWell we could do although you would have then have to sign the assembly as it is currently not strongly named and then install it in the GAC with gacutil, and as this is still an early beta so I\u0026rsquo;m happy to keep the assembly out of the GAC for now.\nYou should be ready to put MonoGame to use now. If you run the template project you will get a MonoGame logo drawn to the screen. I suggest looking at some of the demo\u0026rsquo;s in the Samples folder and maybe try porting a couple of them over to F#. In future posts Ill try to address some of the functional aspect of using F# with MonoGame.\nWhat\u0026rsquo;s next? # I\u0026rsquo;m currently starting work on a post which shows F# using some of the 3D aspects of MonoGame. Exciting!\nUntil next time\u0026hellip;\n","date":"November 11, 2012","externalUrl":null,"permalink":"/programming/2012-11-11-fsharp-and-monogame-on-the-mac/","section":"Blog","summary":"What we are going to do in this post is take a whistle stop tour of getting MonoGame up and running along with a simple demo in F#. Over the last few days I have been building an F# project template for MonoDevelop, this post will also how to get that installed too. ","title":"F# and MonoGame on the Mac","type":"programming"},{"content":"So what have I been up to lately? Well, lots of different things. I have been taking it easy on the open source and blogging side of things as its been a hectic time of late in my personal life. This seems to be changing now so I\u0026rsquo;m starting to get all of the ideas spinning around in my head into physical reality, or virtual reality, or what ever you want to call it. Anyway, here\u0026rsquo;s the first post on the subject of programming in F# using MonoDevelop and Mono natively on Macs. I have always been a fan of Mono but I have always shied away from using it in anger, this is mainly due to my Windows heritage and using Visual Studio almost exclusively at work. Times are changing though, and over the past few years I have been using Macs more and more. In fact the only place where I still use Windows natively is at work, so it makes sense to get a development environment up and running to support me.\nOnce the Beta of Mono 3.0 goes live it should contain within it a nice shiny new F# 3.0 installation, but until then we have to do a number of manual steps:\nInstall Mono 3.0 Install MonoDevelop Compile and install F#3.0 Compile and install the MonoDevelop F# Language Binding Install Mono 3.0 # Head over to the Mono site and install Mono 3.0 and the Mono MDK, they can be both found on the download page.\nIF you think you already have these installed you can check which version you have by Navigating to the directory: /Library/Frameworks/Mono.framework/Versions You should see version 3.0 in the list.\nAlso ensure that the Current symbolic link points to Mono version 3.0, when I was first tried to get things running I couldn\u0026rsquo;t understand why I kept getting a version mismatches, this was because my link was pointing to Mono 2.10.9.\nInstall MonoDevelop # I would recommend installing MonoDevelop straight from the packaged installer, as I write this the stable version is 3.0.4.7. You can find the download page here.\nBuilding F#3.0 # These instruction can also be found here but I thought it might be helpful to keep everything at hand here, if nothing else it will serve as a reference if I need to do this again or help anybody out.\n#Make sure you have automake installed: brew install automake #First of all clone the F#3.0 repo from GitHub git clone git://github.com/fsharp/fsharp.git\\ #change the directory cd fsharp #run autogen script pointing at the current version of Mono ./autogen.sh --prefix=/Library/Frameworks/Mono.framework/Versions/Current/ #run make to compile the code make #run make to install sudo make install Once that\u0026rsquo;s all finished you can check everything is working by running fsharpc:\nfsharpc F# Compiler for F# 3.0 (Open Source Edition) Freely distributed under the Apache 2.0 Open Source License error FS0207: No inputs specified If you didn\u0026rsquo;t get a version 3.0 Compiler build displayed then check where fsharpc is running from by using the type command:\ntype fsharpc fsharpc is /usr/bin/fsharpc You can now navigate there and check where the symbolic link is pointing to be using the -l parameter of the ls command:\ncd /usr/bin ls -l fsharpc lrwxr-xr-x 1 root wheel 51 3 Nov 09:33 fsharpc -\u0026gt; /Library/Frameworks/Mono.framework/Commands/fsharpc When I first tried to get the F# 3.0 compiler installed I had all sorts of problem with the symbolic link pointing to the old version. I don\u0026rsquo;t know whether the installer had failed or whether an old version of the F# compiler or the F# language binding had caused issues, the process was so much smoother on my MacBook Pro as it was a clean install.\nBuild the MonoDevelop F# language binding # git clone git://github.com/fsharp/fsharpbinding.git cd fsharpbinding ./configure.sh make make install That\u0026rsquo;s it we\u0026rsquo;re all done! Pretty easy stuff thanks to Ben Winkel and others for putting in some time to fix up the issues.\nNow you should be able to fire up MonoDevelop and start building some F# code!\nYou can check the add-in is installed properly my opening the Add-in Manager from the MonoDevelop menu. The F# language binding should be shown in the list.\nWhen you start a new solution you should now be presented with some F# specific options.\nNext time I will take this a step further by showing how to build, install and integrate MonoGame with F#. I will also be releasing an F# specific project template for use with MonoDevelop.\nUntil next time\u0026hellip;\n","date":"November 3, 2012","externalUrl":null,"permalink":"/programming/2012-11-03-fsharp-3-in-the-mac-and-mono-world/","section":"Blog","summary":"So what have I been up to lately? Well, lots of different things. I have been taking it easy on the open source and blogging side of things as its been a hectic time of late in my personal life. This seems to be changing now so I’m starting to get all of the ideas spinning around in my head into physical reality, or virtual reality, or what ever you want to call it. Anyway, here’s the first post on the subject of programming in F# using MonoDevelop and Mono natively on Macs. ","title":"F# 3.0 In The Mac And Mono World","type":"programming"},{"content":"If I walk into my garage now and open up a toolbox, whats inside?\nHere\u0026rsquo;s a quick selection:\nBall-peen hammer Jointer plane 1/2 inch mortise chisel Soldering iron Set square Low angle block plane Torx screw drivers Hack saw Monkey wrench Pipe cutter Notice it doesn\u0026rsquo;t just contain:\nA sledge hammer. Different tools have different purposes, you wouldn\u0026rsquo;t use a hammer and try to cut down a tree, or use a chisel to hammer a nail.\nSoftware # In the software industry we also have access to a vast array of different tools all for different purposes: Image editors, text editors, social media clients, email clients, it\u0026rsquo;s endless! Most of the time we pick the right tool for the job. You wouldn\u0026rsquo;t use notepad or TextMate to edit an image file, although you could, you just wouldn\u0026rsquo;t.\nProgramming # The same can be said about programming languages. Programming languages fall into different styles, no single programming style is best suited to solve every problem.\nThere are four main styles or paradigms:\nImperative - The imperative style spells out the problem in detail, the programmer explicitly defined the exacting steps the program must follow. Functional - Functional programming favors immutability and functional composition, the programmer thinks about the problem as sequence of stateless function evaluations. Logic - based on the idea of using logical sentences to represent programs and to perform computations. Object-oriented - The programmer thinks about the problem as a collection of interacting objects. Some languages are said to be multi-paradigm, supposedly allowing you to use you the right tool for the job. This only helps to a certain extent as most languages embrace a single paradigm more than others. In addition, there are also classes of problems that cannot be easily solved simply with one of the main paradigms. Problems such as distributed communication and fault tolerance which I will touch on later.\nA bit of history # Early on in my career I worked solely in C#. For years I toiled away using ASP.Net, WinForms, ADO.Net, etc before moving into back-end server-side problems. I always found myself solving problems which other people shied away from, like multi-threading and performance profiling, so I spent another few years heavily engaged in server-side projects.\nSeveral years ago a colleague and I were brought in to fix an existing application which had to be scaled out to support thousands of concurrently connected clients. It was written in C# to replace an aging C++ version which had also failed to deliver. We found ourselves rapidly rewriting most of the core application from the inside out. We developed a solid core framework before proceeding any further.\nI have always found that following design principles like SOLID rather than blindly following patterns results in a well rounded solution. The design that followed had a distinct functional nature, although we didn\u0026rsquo;t know that at the time, in hindsight we had discovered functional programming for ourselves.\nI cant remember exactly how it happened now, but at the time I was reading about Ocaml and F#. This is when I really started to embrace functional programming, I started to tie together the design decisions we had made during the project against the functional programming concepts I was reading about.\nWhere am I now? # Now that I have a better awareness of different styles I can more readily think through the problem at hand and tailor the solution in the right direction. I\u0026rsquo;m now also using Clojure, Erlang, and Haskell. Failure to at least be aware of different languages, what they are good at, and what problems they effectively solve is a big mistake in my eyes. I try to do everything to the best of my ability, if your going to do something you might as well do it properly!\nThroughout my professional career various problems have came up at different times and by far the most time consuming problems are those in the realm of concurrency, scalability, and fault tolerance. I have often found myself devoting large amounts of time to solving these issues, but there are languages which exist which deal with this at their core level - namely Erlang. This is also something that I have been looking into recently.\nThis blog # This blog will be about using different tools for different jobs. It will also be a place for me to have a rant about things which are bugging me. Don\u0026rsquo;t expect my usual pure F# output, expect posts on: Erlang, Clojure, Haskell, C#, or even, dare I say it, Javascript! Don\u0026rsquo;t worry though, there will still be plenty of F# content too.\nAll the photos in this post are of my own tools taken by my wife Lynsey who is a keen photographer, you can find some more examples of her work on Flickr.\nUntil next time\u0026hellip;\n","date":"August 27, 2012","externalUrl":null,"permalink":"/programming/2012-08-23-whats-in-your-toolbox/","section":"Blog","summary":"If I walk into my garage now and open up a toolbox, whats inside?\nHere’s a quick selection:\nBall-peen hammer Jointer plane 1/2 inch mortise chisel Soldering iron Set square Low angle block plane Torx screw drivers Hack saw Monkey wrench Pipe cutter Notice it doesn’t just contain:\n","title":"Whats in your toolbox?","type":"programming"},{"content":"","date":"July 15, 2012","externalUrl":null,"permalink":"/tags/agents/","section":"Tags","summary":"","title":"Agents","type":"tags"},{"content":"","date":"July 15, 2012","externalUrl":null,"permalink":"/tags/mailboxprocessor/","section":"Tags","summary":"","title":"Mailboxprocessor","type":"tags"},{"content":"Deep in the darkest depths lurks an ancient horror, when the time is right it will rise forth and leave you screaming for mercy and begging for forgiveness\u0026hellip;\nOK, I have a penchant for being over dramatic but in this post I am going to reveal some little known caveats in a well known and much revelled area of F#, agents aka the MailboxProcessor. Gasp!\nFirst let me give you a demonstration:\nopen System open System.Diagnostics type internal BadAgentMessage = | Message of string * int | Lock | Unlock type BadAgent() = let agent = MailboxProcessor.Start(fun agent -\u0026gt; let sw = Stopwatch() let rec waiting () = agent.Scan(function | Unlock -\u0026gt; Some(working ()) | _ -\u0026gt; None) and working() = async { let! msg = agent.Receive() match msg with | Lock -\u0026gt; return! waiting() | Unlock -\u0026gt; return! working() | Message (msg, iter) -\u0026gt; if iter = 0 then sw.Start() if iter % 10000 = 0 then sw.Stop() printfn \u0026#34;%s : %i in: %fms\u0026#34; msg iter sw.Elapsed.TotalMilliseconds sw.Restart() return! working() } working()) member x.Msg(msg) = agent.Post(Message msg) member x.Lock() = agent.Post(Lock) member x.Unlock() = agent.Post(Unlock) The BadAgentMessage type defines a discriminated union that we are going to use for the agents message interface. This is comprised of three elements:\nMessage: This will just be a simple string-based message and an int used as a counter. Lock: This is used to stop message processing within the agent by causing it to wait for an Unlock message to arrive. Unlock: This message is used to resume the processing within the agent, effectively exiting the locked state. We have two main sections to the agents body which I will describe below.\nworking # The purpose of the working function is to dequeue the messages from the agent and process them with pattern matching; let! msg = agent.Receive() is used to get the next message which is then pattern matched to be one of the three messages types of the BadAgentMessage. When the Lock message is encountered return! waiting() is used to place the agent in a state where it is waiting for an Unlock message to arrive. An Unlock message simply resumes processing by calling return! working(). The only real purpose of the Unlock message is to exit from the locked state that is introduced by the Lock message. The Message message simply starts a StopWatch on the first operation by using the Messages counter, and then stops it again on the 10,000th operation. At this point the time taken is also printed to the console and the StopWatch is restarted before resuming the main processing loop by calling return! working()\nwaiting # This function is using the agents Scan function to wait for an Unlock message to arrive, once it does it puts the agent back into normal operation by calling returning Some(working()) from the Scan function. If the message does not match an Unlock message then None is returned and the agent simply waits for the next message before trying again.\nThe rest of the agent is just ancillary member functions to allow easy sending of the three message types.\nTest Harness # And here\u0026rsquo;s a very simple test harness:\nlet ba = BadAgent() printfn \u0026#34;Press and key to start\u0026#34; Console.ReadLine() |\u0026gt; ignore let dump number = for i in 0 .. number do ba.Msg(\u0026#34;A message\u0026#34;, i) ta.Lock() dump 200000 ta.Unlock() Console.ReadLine() |\u0026gt; ignore OK, so this is a very synthetic test but I just wanted to highlight some of the internal behaviour. If I run this code I get the following console output:\nYou can see that the time to process the first 10,000 messages is 3083ms then it steadily decreases until the last 10,000 messages are processed in 94ms. The processing time for 10,000 messages is about 33 times slower at the beginning than as it is at the end. Why?\nOpening it up # Let\u0026rsquo;s take a look at some of the internals of the MailboxProcessor to understand what\u0026rsquo;s going on. First of all the core functionality is actually contained within the Mailbox type with the MailboxProcessor acting as an augmenter. TryPostAndReply, PostAndReply, PostAndTryAsyncReply, and PostAndAsyncReply all add a single functionality to the Mailbox type; the ability to synchronously or asynchronously reply to a message once it arrives. TryPostAndReply and PostAndReply both wait synchronously for a message to arrive before replying, whereas PostAndTryAsyncReply and PostAndAsyncReply both reply asynchronously. This functionality is achieved with the use of the ResultCell and AsyncReplyChannel types. For an in-depth discussion on this you might want to refer to my earlier series which describes implementing the MailboxProcessor with TPL Dataflow (see Part 1, Part 2 and Part 3).\nBelow are some snippets of code from the Mailbox type you might want to take a peek yourself at the FSharp repository over at Github for a closer inspection, be warned thought there is a lot of code in there!\nHere\u0026rsquo;s the initial type definition for the Mailbox, you can see that there are two mutable fields:\ntype Mailbox\u0026lt;\u0026#39;Msg\u0026gt;() = let mutable inboxStore = null let mutable arrivals = new Queue\u0026lt;\u0026#39;Msg\u0026gt;() inboxStore is a generic List type System.Collection.Generic.List\u0026lt;T\u0026gt; and arrivals is a System.Collections.Generic.Queue\u0026lt;T\u0026gt; type.\nFor now the inboxStore is null and is only ever assigned via Scan or TryScan and this is done indirectly via the inbox member shown here:\nmember x.inbox = match inboxStore with | null -\u0026gt; inboxStore \u0026lt;- new System.Collections.Generic.List\u0026lt;\u0026#39;Msg\u0026gt;(1) // ResizeArray | _ -\u0026gt; () inboxStore Understanding the code in the Mailbox can be difficult given the amount of code, so I\u0026rsquo;ll highlight the key functions in the sections below to make it a little easier.\nScan / TryScan # Scan is just an async wrapper around TryScan. If TryScan returns None an exception is raised, if not then the result from TryScan is returned.\nSo now lets take a look at the source of TryScan.\nmember x.TryScan ((f: \u0026#39;Msg -\u0026gt; (Async\u0026lt;\u0026#39;T\u0026gt;) option), timeout) : Async\u0026lt;\u0026#39;T option\u0026gt; = let rec scan() = async { match x.scanArrivals(f) with | None -\u0026gt; // Deschedule and wait for a message. When it comes, rescan the arrivals let! ok = waitOne(timeout) if ok then return! scan() else return None | Some resP -\u0026gt; let! res = resP return Some(res) } // Look in the inbox first async { match x.scanInbox(f,0) with | None -\u0026gt; return! scan() | Some resP -\u0026gt; let! res = resP return Some(res) } You can see here that an async workflow is declared that first pattern matches on x.scanInbox, passing in the predicate scan function f and the literal 0. If None is returned then there is no match and the recursive function scan is returned. This time the function x.scanArrivals is be called, again passing in the predicate function f.\nAn interesting point to note, is that each message that arrives that doesn\u0026rsquo;t match the predicate f resets the timer: let! ok = waitOne(timeout), this means that any number of trivial messages that arrive keep the TryScan function running. This was also mentioned by Jon Harrop in a Stackoverflow question entitled How to use TryScan in F# properly. Jon also mentions locking which I will address in the scanArrivals section below. So what\u0026rsquo;s the difference between scanArrivals and scanInbox?\nscanInbox operates on the inboxStore which you might recall is a List\u0026lt;T\u0026gt; type, whereas scanArrivals operates on arrivals which is a Queue\u0026lt;T\u0026gt; type. The big difference between these two is that as messages first arrive in the Mailbox they end up in the arrivals queue first, and when messages are not matched by the predicate function f they are added to the inboxStore, hence the need to always check the inboxStore before the arrivals queue otherwise previously unmatched scan messages would not be processed correctly. You might be asking yourself why not use a Queue\u0026lt;T\u0026gt; for both the inbox and the arrivals? It comes down to the fact that it\u0026rsquo;s not possible to easily use a Queue\u0026lt;T\u0026gt; for arrivals because of the way that Scan works. At any point in the queue there could do a potential match so each item would have to be dequeued and processed separately, an indexed List\u0026lt;T\u0026gt; type is the best fit for this situation.\nscanArrivals / scanArrivalsUnsafe # Lets look at the scanArrivals function, it\u0026rsquo;s just a lock construct around the scanArrivals function. This leads to an important point, the scan function is operating under a lock, which effectively means that end user code is also executed under the lock and if you hold onto the lock for any length of time then there will be significant blocking of the normal receive mechanism due to it also using the same lock when receiving.\nmember x.scanArrivalsUnsafe(f) = if arrivals.Count = 0 then None else let msg = arrivals.Dequeue() match f msg with | None -\u0026gt; x.inbox.Add(msg); x.scanArrivalsUnsafe(f) | res -\u0026gt; res // Lock the arrivals queue while we scan that member x.scanArrivals(f) = lock syncRoot (fun () -\u0026gt; x.scanArrivalsUnsafe(f)) If we pause for a second and review the MailBoxProcessor documentation on MSDN:\nFor each agent, at most one concurrent reader may be active, so no more than one concurrent call to Receive, TryReceive, Scan or TryScan may be active.\nObeying this rule should ensure that no deadlock situations will arise but lock contentions can still arise as messages will still be being posted to the mailbox, which will in turn attempt to acquire the same syncRoot lock.\nLets move onto the next function, I have saved this one for last as its the most interesting.\nscanInbox # A quick glance at scanInbox reveals another function which, to my eye, could have heavy-weight performance implications. The inbox is a List\u0026lt;T\u0026gt; type, and the RemoveAt function does an internal Array.Copy for each removal. This is an O(n) operation where n is (Count - index), so as soon as the list gets to a reasonable size then this then is going to really start chewing into your processing time.\nmember x.scanInbox(f,n) = match inboxStore with | null -\u0026gt; None | inbox -\u0026gt; if n \u0026gt;= inbox.Count then None else let msg = inbox.[n] match f msg with | None -\u0026gt; x.scanInbox (f,n+1) | res -\u0026gt; inbox.RemoveAt(n); res In order to check this theory lets do some quick profiling of the console test that we showed earlier:\nThis screen shot was taken using Jet Brains DotTrace 5.1. This is one of my favourite performance profilers because it captures results to line level and maps back to the F# source code relatively easily.\nYeah there it is, a whopping 44.41% of the time is spent in RemoveAt. Also notice that there were 200,000 calls which mirrors the number we placed in the queue before using the Lock/Unlock message types.\nOne of the things that really stands out for me is that the inbox is a simple list and completely unbounded. In a high throughput situation where the scan function is being used it\u0026rsquo;s perfectly feasible to get into a runaway memory or CPU condition where the unmatched messages are sitting in the inbox taking longer and longer to processes due to the O(n) operation that takes place in the RemoveAt function. Given a consistent throughput then eventually you are going to either run out memory, or the processing time will make throughput drop to dire levels which in turn will back up the inbox even further, effectively this is a death spiral.\nConclusion # So what conclusion can we draw from all of this?\nFirstly be careful with usage of Scan and TryScan, in certain situations the internal queue could back up to a certain size where you will be constantly struggling against the O(n) operation cost. Agents are not a silver bullet solution. They cannot solve every problem. Although it\u0026rsquo;s possible to use agent based techniques to solve various problems like blocking collections and such like, you have to use care and diligence in the solution to avoid introducing another problems into the mix. I have seen several implementations that I have been able to break relatively easily. Do I still use agents? Absolutely! Agents are a fabulous tool to have in our toolbox and some extremely elegant solution exist to solve very complex problems. Do I use Scan or TryScan? Not in its current form in the MailboxProcessor. I chose to implement a destructive scan in my TDF agent for the reasons discussed here. Before we finish, I\u0026rsquo;d like to briefly cover TryScan from my TDF based agent to complete the picture.\nDestructive TryScan # member x.TryScan((scanner: \u0026#39;Msg -\u0026gt; Async\u0026lt;_\u0026gt; option), timeout): Async\u0026lt;_ option\u0026gt; = let ts = TimeSpan.FromMilliseconds(float timeout) let rec loopForMsg = async { let! msg = Async.AwaitTask \u0026lt;| incomingMessages.ReceiveAsync(ts) .ContinueWith(fun (tt:Task\u0026lt;_\u0026gt;) -\u0026gt; if tt.IsCanceled || tt.IsFaulted then None else Some tt.Result) match msg with | Some m -\u0026gt; let res = scanner m match res with | None -\u0026gt; return! loopForMsg | Some res -\u0026gt; return! res | None -\u0026gt; return None} loopForMsg A message is dequeued on the line 4 with let! msg = Async.AwaitTask .... This is then processed by the pattern matching expression on line 9 | Some m -\u0026gt; let res = scanner m. If the result of the scanner function results in None being returned then the message is discarded and the next operation continues with another call to loopForMsg, otherwise the message is returned with | Some res -\u0026gt; return! res.\nOne of the areas where I have a lot of experience is using pipelined operations based on input from network I/O. One of the things that always causes a problem is unbounded situations such as having a queue with no absolute limit. There comes a time when you have to protect yourself from what is effective a denial of service, you have to either destructively terminate messages or connections or route the overflowed data for processing later.\nUntil next time\u0026hellip;\n","date":"July 15, 2012","externalUrl":null,"permalink":"/programming/2012-07-15-the-lurking-horror/","section":"Blog","summary":"Deep in the darkest depths lurks an ancient horror, when the time is right it will rise forth and leave you screaming for mercy and begging for forgiveness…\nOK, I have a penchant for being over dramatic but in this post I am going to reveal some little known caveats in a well known and much revelled area of F#, agents aka the MailboxProcessor. Gasp!\n","title":"The Lurking Horror","type":"programming"},{"content":"In the last post I discussed an asynchronous version of the ManualResetEvent and as promised this time we will be looking at an asynchronous version of the AutoResetEvent. I\u0026rsquo;m using Stephen Toubs post as reference and we will be building a version that is functional in style that maps straight into asynchronous work flows without and conversion or adaptors.\nWhat is an AutoResetEvent? # An AutoResetEvent can be described as a turnstile mechanism, it lets a single waiting person through before re-latching waiting for the next signal. This is opposed to a ManualResetEvent which functions like an ordinary gate. Calling Set opens the gate, allowing any number of threads that are waiting to be let through. Calling Reset closes the gate.\nAsyncAutoResetEvent # First of all here is the shape of the type that we will be building:\ntype AsyncAutoResetEvent = new : ?reusethread:bool -\u0026gt; AsyncAutoResetEvent member Set : unit -\u0026gt; unit member WaitAsync : unit -\u0026gt; Async\u0026lt;bool\u0026gt; Fairly simple: implied constructor, Set and WaitAsync members.\nImplied Constructor # Thinking about this logically we may need the following items:\nA queue mechanism to store asynchronous waiters - let mutable awaits = Queue\u0026lt;_\u0026gt;(). A way of knowing if a signal has been made in the absence of any waiters - let mutable signalled = false. We can also declare a short-circuit asynchronous workflow for the situation that Set() is called before WaitAsync() let completed = async.Return true. This will save us constructing an AsyncResultCell\u0026lt;_\u0026gt; and going though the rest of the asynchronous mechanism. Also notice that an optional parameter called reusethread is defined, we use the ? prefix when defining it to make it optional. We then make use of the defaultArg function to give it a default value of false if a one is not passed in. This will be used in the Set operation to determine if the code will run on the same thread or a thread in the ThreadPool.\nopen System open System.Threading open System.Collections.Generic type AsyncAutoResetEvent(?reusethread) = let mutable awaits = Queue\u0026lt;_\u0026gt;() let mutable signalled = false let completed = async.Return true let reuseThread = defaultArg reusethread false WaitAsync() # The first step is to use a locking construct to control access to the mutable queue awaits. Inside this lock we check to see if signalled is true and if so we reset it to false and return our pre-built completed asynchronous workflow. If signalled is false then we create a new AsyncResultCell\u0026lt;_\u0026gt; and add it to the queue then return the AsyncResult to the caller.\nmember x.WaitAsync() = lock awaits (fun () -\u0026gt; if signalled then signalled \u0026lt;- false completed else let are = AsyncResultCell\u0026lt;_\u0026gt;() awaits.Enqueue are are.AsyncResult) Set() # We first declare a function called getWaiter(), we use this function to return an option type that is either Some AsyncResultCell\u0026lt;bool\u0026gt; or None. We use the lock function to control access to the mutable queue lock awaits. Once inside the lock we use pattern matching to capture awaits.Count and signalled:\nThe first pattern match (x,_) checks if there are any waiters (awaits.Count \u0026gt; 0) and then dequeues an AsyncResultCell\u0026lt;bool\u0026gt; from the queue and returns it within an option type: Some \u0026lt;| awaits.Dequeue(). The second pattern match (_,y) checks whether signalled is set to false before setting its value to true. This causes next WaitAsync() caller to get the short-circuited value completed. This means that an AsyncResultCell\u0026lt;bool\u0026gt; does not need to be created and go though the whole async mechanism. We then return None as there is no waiter to be notified. The final pattern match (_,_) is used when there are no waiting callers and signalled has already being set, there is simply nothing to do in this situation so we return None. We use the getWaiter() function via pattern match. If we have a result i.e. Some AsyncResultCell then we call RegisterResult passing in AsyncOK(true) to indicate a completion. Notice that we also pass in the reuseThread boolean that was declared as part of the constructor. If reuseThread is true then the notification to the waiter happens synchronously use this with care! Personally I would stick with the default of false to ensure that the operation is completed via the thread pool, unless you have a performance critical reason and the waiting code that executes is very fast.\nmember x.Set() = let getWaiter()= lock awaits (fun () -\u0026gt; match (awaits.Count, signalled) with | (x,_) when x \u0026gt; 0 -\u0026gt; Some \u0026lt;| awaits.Dequeue() | (_,y) when not y -\u0026gt; signalled \u0026lt;- true;None | (_,_) -\u0026gt; None) match getWaiter() with | Some a -\u0026gt; a.RegisterResult(AsyncOk(true), reuseThread) | None _ -\u0026gt; () The reason for using the getWaiter() function is to separate the locking function away from the notification, if RegisterResult was called within the lock and reuseThread was true then the awaiting function would be called synchronously within the lock which would not be a very good situation to be in.\nSo there we have it, I could take this series further and convert the other primitives that Stephen Toub describes but there should be enough information in these two posts to set you on your way. If anyone would like me to complete the series then let me know. I may well finish them off and post them on GitHub in the future, time permitting.\nThanks for tuning in, until next time\u0026hellip;\n","date":"April 22, 2012","externalUrl":null,"permalink":"/programming/2012-04-22-back-to-the-primitive-ii/","section":"Blog","summary":"In the last post I discussed an asynchronous version of the ManualResetEvent and as promised this time we will be looking at an asynchronous version of the AutoResetEvent. I’m using Stephen Toubs post as reference and we will be building a version that is functional in style that maps straight into asynchronous work flows without and conversion or adaptors.\n","title":"Back to the Primitive II","type":"programming"},{"content":"","date":"April 22, 2012","externalUrl":null,"permalink":"/series/backtotheprimitive/","section":"Series","summary":"","title":"Backtotheprimitive","type":"series"},{"content":"","date":"April 22, 2012","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"","date":"April 22, 2012","externalUrl":null,"permalink":"/tags/threading/","section":"Tags","summary":"","title":"Threading","type":"tags"},{"content":"","date":"April 22, 2012","externalUrl":null,"permalink":"/tags/tpl/","section":"Tags","summary":"","title":"Tpl","type":"tags"},{"content":"In this post we are going back to the primitive. No it\u0026rsquo;s not about the same named song by Soulfly, (which incidentally does contains F# notes) but a return to thread synchronisation primitives and their asynchronous counterparts.\nWe are going to be looking at an asynchronous version of the ManualResetEvent. This was recently covered by Stephen Toub on the pfx team blog. We will be taking a slightly different view on this as we will be using asynchronous workflows which will give us nice idiomatic usage within F#.\nFirst lets look of the shape of the type that Stephen defined:\npublic class AsyncManualResetEvent { public Task WaitAsync(); public void Set(); public void Reset(); } Now this can be used from within F# by using the Async.AwaitTask function from the Async module but this is like wrapping one asynchronous paradigm with another, and although this does work, what if you want to avoid the overhead of wrappers and stay strictly within async workflows.\ntype asyncManualResetEvent() = member x.WaitAsync() : unit -\u0026gt; Async\u0026lt;bool\u0026gt; member x.Set() : unit -\u0026gt; unit member x.Reset() : unit -\u0026gt; unit That\u0026rsquo;s what we want to see! I don\u0026rsquo;t want to get into the details of the description of how the C# version works as Stephen does a very good job of that already. What I will explain though is how we essentially do the same thing while staying with the realm of functional programming. As we are getting into the lower lever details no doubt we will have to start relying on some low level locking primitives like Monitors, Semaphores, and Interlocked operations, even the F# core libraries have a cornucopia of those.\nLets look at the first member WaitAsync(). The first step is to create a something to store the result of the operation, all we will just be storing and returning asynchronously is a boolean to indicate that the wait handle has been set. To do this we use one of the types from the F# power pack AsyncResultCell\u0026lt;'T\u0026gt;. I think that such a type should of been exposed from the F# core libraries but it was omitted for some reason. There is a type called ResultCell\u0026lt;'T\u0026gt; with much the same functionality in the FSharp.Core.Control namespace but it is marked internal so it\u0026rsquo;s not available for our use.\nWe declare a reference cell of type AsyncResultCell\u0026lt;'T\u0026gt; and then create the WaitAsync() member, all we have to do is dereference the value of the reference cell with ! and call its AsyncResult member, this gives us an Async\u0026lt;bool\u0026gt; which we can easily use in an asynchronous workflow.\ntype asyncManualResetEvent() = let aResCell = ref \u0026lt;| AsyncResultCell\u0026lt;_\u0026gt;() member x.WaitAsync() = (!aResCell).AsyncResult The next bit is fairly simple too. All we need to do is dereference the value of the reference cell, and invoke the RegisterResult member by passing in a value of AsyncOk(true). The boolean value of true will be used by the type inference system to constrain the value of the Async\u0026lt;_\u0026gt; returned from WaitAsync.\nmember x.Set() = (!aResCell).RegisterResult(AsyncOk(true)) The last part is the most complex (as usual). Here we create a recursive function called swap that will try to exchange the AsyncResultCell\u0026lt;'T\u0026gt; for a new one. We dereference the reference cell to currentValue, then we use a CAS (Compare And Swap) operation to compare the aResCell with currentValue and if they are equal newVal will replace aResCell. On the next line if the result of the CAS operation means that result and currentValue are equal then we are finished, otherwise we spin the current thread for 20 cycles using Thread.SpinWait 20 before retrying the operation via recursion swap newVal. This will be a lot less expensive than switching to user or kernel mode locking, and the period of contention between threads should be very small. Finally the swap operation is started by passing in a new AsyncResultCell\u0026lt;'T\u0026gt;.\nThere are various other methods we could of used, for instance we could of wrapped a ManualResetEvent with a call to Async.AwaitWaitHandle, although this would of meant using the kernel mode locking of the ManualResetEvent which is a bit more expensive.\nIn Stephen Toub\u0026rsquo;s post he mentions Task\u0026rsquo;s being orphaned due to the Reset() method being called before the Task\u0026lt;'T\u0026gt; has been completed, that shouldn\u0026rsquo;t happen in our implementation due the the closures being stored internally for completion by the async infrastructure. Heres a quick test harness to make sure everything works as expected anyway.\nmember x.Reset() = let rec swap newVal = let currentValue = !aResCell let result = Interlocked.CompareExchange\u0026lt;_\u0026gt;(aResCell, newVal, currentValue) if obj.ReferenceEquals(result, currentValue) then () else Thread.SpinWait 20 swap newVal swap \u0026lt;| AsyncResultCell\u0026lt;_\u0026gt;() let amre = asyncManualResetEvent() let x = async{let! x = amre.WaitAsync() Console.WriteLine(\u0026#34;First signalled\u0026#34;)} let y = async{let! x = amre.WaitAsync() Console.WriteLine(\u0026#34;Second signalled\u0026#34;)} let z = async{let! x = amre.WaitAsync() Console.WriteLine(\u0026#34;Third signalled\u0026#34;)} //start async workflows x and y Async.Start x Async.Start y //reset the asyncManualResetEvent, this will test whether the async workflows x and y // are orphaned due to the AsyncResultCell being recycled. amre.Reset() //now start the async z Async.Start z //we set a single time, this should result in the three async workflows completing amre.Set() Console.ReadLine() |\u0026gt; ignore Here we can see everything works out as we expected:\nThats all there is too it, next time I will be exploring an asyncAutoResetEvent in much the same vein.\nUntil next time\u0026hellip;\n","date":"April 12, 2012","externalUrl":null,"permalink":"/programming/2012-04-12-back-to-the-primitive/","section":"Blog","summary":"In this post we are going back to the primitive. No it’s not about the same named song by Soulfly, (which incidentally does contains F# notes) but a return to thread synchronisation primitives and their asynchronous counterparts.\nWe are going to be looking at an asynchronous version of the ManualResetEvent. This was recently covered by Stephen Toub on the pfx team blog. We will be taking a slightly different view on this as we will be using asynchronous workflows which will give us nice idiomatic usage within F#.\n","title":"Back to the Primitive","type":"programming"},{"content":"In this edition we are going to be doing a taste test, C# vs F#. Oh yeah, if you quickly glanced at the title you may have thought this was a recipe for black scones, as interesting and tasty as that may be, unfortunately its going to be finance related.\nI recently presented a paper on the benefits of F#, part of this was a comparison of the famous Black-Scholes equation in both C# and F#. I was mainly going to be looking at code succinctness and the inherent suitability of the language for calculation based work, but there ended up being more to it than that.\nFirst of all I quickly set up a test rig to run 50 million iterations of the algorithm to see if there were any difference in the processing speed. I want expecting any major differences at this point but here\u0026rsquo;s what I got:\nC# results for 50 million iterations F# results for 50 million iterations I think you will agree that\u0026rsquo;s quite a difference, lets have a look at the code to see what\u0026rsquo;s going on.\nC# Implementation # public class Options { public enum Style { Call, Put } public static double BlackScholes(Style callPut, double s, double x, double t, double r, double v) { double result = 0.0; var d1 = (Math.Log(s / x) + (r + v * v / 2.0) * t) / (v * Math.Sqrt(t)); var d2 = d1 - v * Math.Sqrt(t); switch (callPut) { case Style.Call: result = s * Cnd(d1) -x * Math.Exp(-r * t) * Cnd(d2); break; case Style.Put: result = x * Math.Exp(-r * t) * Cnd(-d2) -s * Cnd(-d1); break; } return result; } private static double Cnd(double x) { const double a1 = 0.31938153; const double a2 = -0.356563782; const double a3 = 1.781477937; const double a4 = -1.821255978; const double a5 = 1.330274429; var l = Math.Abs(x); var k = 1.0 / (1.0 + 0.2316419 * l); var w = 1.0 - 1.0 / Math.Sqrt(2 * Math.PI) * Math.Exp(-l * l / 2.0) * (a1 * k + a2 * k * k + a3 * Math.Pow(k, 3) + a4 * Math.Pow(k, 4) + a5 * Math.Pow(k, 5)); if (x \u0026lt; 0) { return 1.0 - w; } return w; } } F# Implementation # module options open System type Style = Call | Put let cnd x = let a1 = 0.31938153 let a2 = -0.356563782 let a3 = 1.781477937 let a4 = -1.821255978 let a5 = 1.330274429 let l = abs x let k = 1.0 / (1.0 + 0.2316419 * l) let w = (1.0 - 1.0 / sqrt(2.0 * Math.PI) * exp(-l * l / 2.0) * (a1 * k + a2 * k * k + a3 * (pown k 3) + a4 * (pown k 4) + a5 * (pown k 5))) if x \u0026lt; 0.0 then 1.0 - w else w let blackscholes style s x t r v = let d1 = (log(s / x) + (r + v * v / 2.0) * t) / (v * sqrt(t)) let d2 = d1 - v * sqrt(t) match style with | Call -\u0026gt; s * cnd(d1) -x * exp(-r * t) * cnd(d2) | Put -\u0026gt; x * exp(-r * t) * cnd(-d2) -s * cnd(-d1) Differences # The most significant differences when the code is compiled comes down to a few areas.\nThe BlackScholes function # The first thing to note is the code size and number of local variables:\n// Code size 122 (0x7a) .maxstack 6 .locals init ([0] float64 d1, [1] float64 d2) // Code size 164 (0xa4) .maxstack 4 .locals init ([0] float64 d1, [1] float64 d2, [2] float64 result, [3] valuetype CsBs.Options/Style CS$0$0000) The initial arguments that are loaded in the F# implementation is done in fewer IL op codes then C#.\nIL_0001: ldarg.1 IL_0002: ldarg.2 IL_0003: div IL_0004: call float64 [mscorlib]System.Math::Log(float64) IL_0000: ldc.r8 0.0 IL_0009: stloc.0 IL_000a: ldc.r8 0.0 IL_0013: stloc.1 IL_0014: ldc.r8 0.0 IL_001d: stloc.2 IL_001e: ldarg.1 IL_001f: ldarg.2 IL_0020: div IL_0021: call float64 [mscorlib]System.Math::Log(float64) You can see in the C# code is intialising the local variable to 0.0 by pushing them to the stack ldc.r8 then storing them stloc.0.\nThe pattern matching in the F# code results in a call to get the style options/Style::get_Tag() and then a branch if not equal opcode bne.un.s which causes a jump to IL_005d\nIL_0036: call instance int32 options/Style::get_Tag()``` IL_003b: ldc.i4.1 IL_003c: bne.un.s IL_005d The C# version loads the local variable for the Style IL_0053: stloc.3 and then uses the switch opcode to jump table to jump to either position IL_0064 or IL_0083.\nIL_0053: stloc.3 IL_0054: ldloc.3 IL_0055: switch ( IL_0064, IL_0083) IL_0062: br.s IL_00a2 These are negligible, I\u0026rsquo;m mealy pointing out the differences in compilation between the two languages.\nThe F# compiler is more stringent when compiling the code.\nThe Cnd function # The Cnd function or cumulative normal distribution is where the performance differences occur.\nAgain at initialization you can see the C# version is larger by 41.\n// Code size 213 (0xd5) .maxstack 8 .locals init ([0] float64 l, [1] float64 k, [2] float64 w) // Code size 254 (0xfe) .maxstack 6 .locals init ([0] float64 l, [1] float64 k, [2] float64 w) The C# version initialises all the local variables to 0.0.\nIL_0000: ldc.r8 0.0 IL_0009: stloc.0 IL_000a: ldc.r8 0.0 IL_0013: stloc.1 IL_0014: ldc.r8 0.0 IL_001d: stloc.2 Interestingly the C# compiler optimises out the call to Math.PI * 2 but the F# compiler doesn\u0026rsquo;t.\nIL_003a: ldc.r8 2. IL_0043: ldc.r8 3.1415926535897931 IL_004c: mul IL_0057: ldc.r8 6.2831853071795862 From here everything is identical until we get to the power operator section (Math.Pow in the C# version and pown in F#).\nIL_0089: ldloc.1 IL_008a: ldc.i4.3 IL_008b: call float64 [FSharp.Core]Microsoft.FSharp.Core.Operators/OperatorIntrinsics::PowDouble(float64, int32) In the F# code we are using the pown function which calculates the power to an integer. This is shown in the call to OperatorIntrinsics::PowDouble which uses the value in IL_0089: ldloc.1 and also loads the integer 3 with IL_008a: ldc.i4.3.\nIL_009c: ldloc.1 IL_009d: ldc.r8 3. IL_00a6: call float64 [mscorlib]System.Math::Pow(float64, float64) The C# code is using the standard Math.Pow operator which operates on two float64 numbers. The value of 3 is implicitly converted into a float64 during compilation IL_009d: ldc.r8 3..\nThe final difference is at the end of the function.\nIL_00b8: stloc.2 IL_00b9: ldarg.0 IL_00ba: ldc.r8 0.0 IL_00c3: clt IL_00c5: brfalse.s IL_00d3 IL_00c7: ldc.r8 1. IL_00d0: ldloc.2 IL_00d1: sub IL_00d2: ret IL_00d3: ldloc.2 IL_00d4: ret The F# version uses the clt opcode. This pushes 1 if value one on the stack is less than value two otherwise it pushes 0. There is then a brfalse.s which jumps to location IL_00d3 if the first value on the stack is less than or equal to the second value.\nIL_00b8: stloc.2 IL_00b9: ldarg.0 IL_00e5: ldc.r8 0.0 IL_00ee: bge.un.s IL_00fc IL_00f0: ldc.r8 1. IL_00f9: ldloc.2 IL_00fa: sub IL_00fb: ret IL_00fc: ldloc.2 IL_00fd: ret The C# version uses the bge.un.s to jump to location IL_00fc if the first value on the stack is greater than the second. This is negligible in normal runtime but it is interesting to note the difference between the two.\nConclusion # Wow, there was a lot of IL to get through, I hope you stayed with me!\nAlthough the difference in some areas are negligible, every little counts. The implicit conversion of an integer field to a float64 hides the fact that we were using an optimized integer power function in F#, that\u0026rsquo;s performance increase of 168%! Some other side effects of implicit conversion can also lead to subtle bugs due to truncation and overflow. The other benefits are the compiled code uses less instructions and the source code only uses 25 lines compared to 44 in C#.\nUntil next time!\n","date":"March 11, 2012","externalUrl":null,"permalink":"/programming/2012-03-10-black-scholes-taste-test/","section":"Blog","summary":"In this edition we are going to be doing a taste test, C# vs F#. Oh yeah, if you quickly glanced at the title you may have thought this was a recipe for black scones, as interesting and tasty as that may be, unfortunately its going to be finance related.\n","title":"Black-Scholes Taste Test","type":"programming"},{"content":"","date":"February 20, 2012","externalUrl":null,"permalink":"/tags/concurrency/","section":"Tags","summary":"","title":"Concurrency","type":"tags"},{"content":"","date":"February 20, 2012","externalUrl":null,"permalink":"/series/dataflowagents/","section":"Series","summary":"","title":"Dataflowagents","type":"series"},{"content":"This will be the last post on rebuilding the MailboxProcessor using TDF, here\u0026rsquo;s a quick discussion of the missing pieces\u0026hellip;\nFirst, lets start with the simple ones, these don\u0026rsquo;t really require much discussion.\nDefaultTimeout # let mutable defaultTimeout = Timeout.Infinite member x.DefaultTimeout with get() = defaultTimeout and set(value) = defaultTimeout \u0026lt;- value This simply provides a mutable property using Timeout.Infinite as a default setting.\nCurrentQueueLength # member x.CurrentQueueLength() = incomingMessages.Count Another simple one, this methods uses into the underlying BufferBlock to extract its current queue length using its Count property.\nTryReceive # member x.TryReceive(?timeout) = let ts = TimeSpan.FromMilliseconds(float \u0026lt;| defaultArg time out defaultTimeout) Async.AwaitTask \u0026lt;| incomingMessages.ReceiveAsync(ts) .ContinueWith(fun (tt:Task\u0026lt;_\u0026gt;) -\u0026gt; if tt.IsCanceled || tt.IsFaulted then None else Some tt.Result) Here we get a little help from TPL to apply a continuation on completion using ContinueWith. We use a lambda to return either None, in a time out condition, or Some tt.Result when we successfully receive an item.\nTryPostAndReply # type AsyncResultCell\u0026lt;\u0026#39;a\u0026gt;() = ... member x.TryWaitResultSynchronously(timeout:int) = //early completion check if source.Task.IsCompleted then Some source.Task.Result //now force a wait for the task to complete else if source.Task.Wait(timeout) then Some source.Task.Result else None member x.TryPostAndReply(replyChannelMsg, ?timeout) :\u0026#39;Reply option = let timeout = defaultArg timeout defaultTimeout let resultCell = AsyncResultCell\u0026lt;_\u0026gt;() let msg = replyChannelMsg(new AsyncReplyChannel\u0026lt;_\u0026gt;(fun reply -\u0026gt; resultCell.RegisterResult(reply))) if incomingMessages.Post(msg) then resultCell.TryWaitResultSynchronously(timeout) else None Things get a little more interesting from here on in. Firstly we need to add a new synchronisation member to the AsyncResultCell\u0026lt;'a\u0026gt; type: TryWaitResultSynchronously. We again enlist the help of the TPL primitives to check for the early completion using source.Task.IsCompleted returning the result if it is there, otherwise we use the Task property\u0026rsquo;s Wait method to check the item returns within the time out interval. In the usual manner, Some source.Task.Result is returned or None for a failure.\nPostAndReply # member x.PostAndReply(replyChannelMsg, ?timeout) : \u0026#39;Reply = match x.TryPostAndReply(replyChannelMsg, ?timeout = timeout) with | None -\u0026gt; raise (TimeoutException(\u0026#34;PostAndReply timed out\u0026#34;)) | Some result -\u0026gt; result This one wraps a call to TryPostAndReply with some pattern matching. In the event of a time out None is returned from TryPostAndReply in this instance we raise a TimeoutException otherwise we unwrap the result from the option using | Some result -\u0026gt; result.\nTryScan # member x.TryScan((scanner: \u0026#39;Msg -\u0026gt; Async\u0026lt;_\u0026gt; option), timeout): Async\u0026lt;_ option\u0026gt; = let ts = TimeSpan.FromMilliseconds( float timeout) let rec loopForMsg = async { let! msg = Async.AwaitTask \u0026lt;| incomingMessages.ReceiveAsync(ts) .ContinueWith(fun (tt:Task\u0026lt;_\u0026gt;) -\u0026gt; if tt.IsCanceled || tt.IsFaulted then None else Some tt.Result) match msg with | Some m -\u0026gt; let res = scanner m match res with | None -\u0026gt; return! loopForMsg | Some res -\u0026gt; return! res | None -\u0026gt; return None} loopForMsg This one also uses the same ContinueWith functionality in the recursive loopForMsg function, perhaps some of these functions could extracted out and refactored but I prefer to keep the code like this to better explain what\u0026rsquo;s going on. The the code is available on GitHub anyway so feel free to clean up any detritus and send me a pull request. Again we use pattern matching to keep calling the loopForMsg function until the result is returned or a time out occurs.\nScan # member x.Scan(scanner, timeout) = async { let! res = x.TryScan(scanner, timeout) match res with | None -\u0026gt; return raise(TimeoutException(\u0026#34;Scan TimedOut\u0026#34;)) | Some res -\u0026gt; return res } Finally we have Scan, this is much like PostAndReply in that it just acts as a wrapper around TryScan making use of pattern matching throwing an exception on a time out.\nThat sums up the last few pieces, completing the TDF implementation of the MailboxProcessor. I think this series of posts has shown the elegance of F#\u0026rsquo;s asynchronous workflows. The use of recursive functions and the compositional nature of asynchronous workflows really helps when you are doing this type of programming. It\u0026rsquo;s also very nice on the eye, each section being clearly defined.\nThe more astute of you may have noticed something a little different. Scan and TryScan are destructive in this implementation, the unmatched messages are purged from the internal queue. Although I could have mirrored the same functionality of the MailboxProcessor by using an internal list to keep track of unmatched messages, this leads to performing checks during Receive and Scan and their derivatives to make sure that this list is used first when switching from Scan and Receive functionality.\nI think the separation of concerns are a little fuzzy in the MailboxProcessor. The scan function seems like an after thought, even if you don\u0026rsquo;t use Scan you still pay a price for it as there are numerous checks between the internal queue and the unmatched messages list. You can also run into issues while using Scan and TryScan that can result in out of memory conditions due to the inherent unbounded nature. I will briefly describe and explore the conditions that can lead to that in the next post. In the implementation presented here we can get bounded checking by passing in an optional DataflowBlockOptions and setting a value for the BoundedCapacity property.\nEDIT: The code for this series of articles is now available on GitHub: FSharpDataflow\nUntil next time\u0026hellip;\n","date":"February 20, 2012","externalUrl":null,"permalink":"/programming/2012-02-19-fsharp-dataflow-agents-iii/","section":"Blog","summary":"This will be the last post on rebuilding the MailboxProcessor using TDF, here’s a quick discussion of the missing pieces…\nFirst, lets start with the simple ones, these don’t really require much discussion.\nDefaultTimeout # let mutable defaultTimeout = Timeout.Infinite member x.DefaultTimeout with get() = defaultTimeout and set(value) = defaultTimeout \u003c- value This simply provides a mutable property using Timeout.Infinite as a default setting.\n","title":"FSharp Dataflow agents III","type":"programming"},{"content":"Right, no messing about this time, straight to the code.\nConstruction # This is pretty straight forward and I don\u0026rsquo;t want to detract from the important bits of this post, the only thing of note is the cancellationToken which is initialized to a default value using the defaultArg function if the optional parameter cancellationToken is not supplied. The TDF construct that we to use to perform most of the hard work is incomingMessages which is a BufferBlock\u0026lt;'Msg\u0026gt;.\ntype DataflowAgent\u0026lt;\u0026#39;Msg\u0026gt;(initial, ?cancellationToken) = let cancellationToken = defaultArg cancellationToken Async.DefaultCancellationToken let mutable started = false let errorEvent = new Event\u0026lt;System.Exception\u0026gt;() let incomingMessages = new BufferBlock\u0026lt;\u0026#39;Msg\u0026gt;() let mutable defaultTimeout = Timeout.Infinite Error # This is the public facing part for the Error event. The [\u0026lt;CLIEvent\u0026gt;] attribute exposes the event in a friendly manner to other .Net languages by adding the add_Error and remove_Error event handler properties to allow subscription to take place. The Error event fires when an exception is thrown in the initial asynchronous workflow.\n[\u0026lt;CLIEvent\u0026gt;] member this.Error = errorEvent.Publish Start # This is implemented the same as the MailboxProcessor. An exception is thrown if the agent has already started as this is not valid operation. We set the mutable field started to true and proceed to start the initial asynchronous workflow. This workflow is wrapped in a try with block so that if an exception is thrown we catch it and trigger the Error event. The computation is then started with Async.Start(...).\nmember this.Start() = if started then raise (new InvalidOperationException(\u0026#34;Already Started.\u0026#34;)) else started \u0026lt;- true let comp = async { try do! initial this with error -\u0026gt; errorEvent.Trigger error } Async.Start(computation = comp, cancellationToken = cancellationToken) Receive # The Receive member is used by the agent as a way of waiting for a message to arrive without blocking. Because the TDF functionality is all TPL Task based we use the the Async helper functions. In this instance we utilise the Async.AwaitTask passing in the incomingMessages ReceiveAsync method to wait for a message to arrive. The integration between F# async and TDF is nice and succinct here.\nmember this.Receive(?timeout) = Async.AwaitTask \u0026lt;| incomingMessages.ReceiveAsync() Post # The Post member allows a message to be sent to the agents, this member simply calls the incomingMessages Post method passing in the item. We raise an exception if there is a problem posting (i.e. the incomingMessages internal queue is full).\nmember this.Post(item) = let posted = incomingMessages.Post(item) if not posted then raise (InvalidOperationException(\u0026#34;Incoming message buffer full.\u0026#34;)) PostAndTryAsyncReply / PostAndAsyncReply # I\u0026rsquo;m grouping both of these together as they are related in functionality. In the previous post I purposely left out some ancillary code as it added unnecessary complexity to the introduction. There are a two types we need to be able to replicate the PostAndTryAsyncReply and PostAndAsyncReply members of the MailboxProcessor.\nAsyncReplyChannel # The first type we need is the AsyncReplyChannel\u0026lt;'Reply\u0026gt;. This type takes a function that accepts a generic 'Reply and returns a unit. It is used as a way of communicating back to the caller of the PostAndTryAsyncReply and PostAndAsyncReply members via its single member Reply. This should become a little clearer when we see it used in context.\nAn AsyncRepyChannel does actually exist in F# under the Microsoft.FSharp.Control namespace and is used my the MailboxPRocessor, unfortunately its constructor is marked as internal so we are not able to reuse it here.\ntype AsyncReplyChannel\u0026lt;\u0026#39;Reply\u0026gt;(replyf : \u0026#39;Reply -\u0026gt; unit) = member x.Reply(reply) = replyf(reply) AsyncResultCell # The next type we need is the AsyncResultCell\u0026lt;'a\u0026gt;. We use this as a way to await for the results of an asynchronous operation. We create a TaskCompletionSource (source), which is a TPL type that we use as a way of signalling to a callback / lambda expression when a message has arrived.\nRegisterResult is used as a way of notifying when a message has been arrived, this is used internally by our agent as a result of a reply being made to the AsyncReplyChannel.\nAsyncWaitResult is a continuation wrapper, it is called when we want to wait indefinitely for the result to be returned. It wraps a successful completion with a call to task.Result which then returns the result.\nGetWaitHandle is used as a mechanism to force the asynchronous result to return within a specified timeout interval. If a result is not returned within the timeout then this function will return false.\nGrabResult returns the result from the TaskCompletionSource object source. This is set earlier by the RegisterResult member.\ntype AsyncResultCell\u0026lt;\u0026#39;a\u0026gt;() = let source = new TaskCompletionSource\u0026lt;\u0026#39;a\u0026gt;() member x.RegisterResult result = source.SetResult(result) member x.AsyncWaitResult = Async.FromContinuations(fun (cont,_,_) -\u0026gt; let apply = fun (task:Task\u0026lt;_\u0026gt;) -\u0026gt; cont (task.Result) source.Task.ContinueWith(apply) |\u0026gt; ignore) member x.GetWaitHandle(timeout:int) = async { let waithandle = source.Task.Wait(timeout) return waithandle } member x.GrabResult() = source.Task.Result PostAndTryAsyncReply # This one is a little more tricky and I have added a few line number references to try and make it easier. On line 3 we declare an resultCell to collect the result of the asynchronous operation. This is used on line 4 when we create a msg to post to incomingMessages on line 5. The replyChannelMsg is a function that takes an AsyncReplyChannel and returns a message, so we create an AsyncReplyChannel with a lambda expression that registers the reply with the resultCell. This is the key to how this works, you have to remember that will be done the other side of the operation which will be within the asynchronous processing loop of the agent when Reply is called on the AsyncReplyChannel.\nFinally pattern matching is used on line 7 to call either AsyncWaitResult or GetWaitHandle on the resultCell. The AsyncWaitResult function is used to wait indefinitely and the GetWaitHandle function is used if we want to use a timeout. Both of these are asynchronous workflows that either return a result or return an option type containing the result.\nmember this.PostAndTryAsyncReply(replyChannelMsg, ?timeout) = let timeout = defaultArg timeout defaultTimeout let resultCell = AsyncResultCell\u0026lt;_\u0026gt;() let msg = replyChannelMsg(AsyncReplyChannel\u0026lt;_\u0026gt;(fun reply -\u0026gt; resultCell.RegisterResult(reply))) let posted = incomingMessages.Post(msg) if posted then match timeout with | Threading.Timeout.Infinite -\u0026gt; async { let! result = resultCell.AsyncWaitResult return Some(result) } | _ -\u0026gt; async { let! ok = resultCell.GetWaitHandle(timeout) let res = (if ok then Some(resultCell.GrabResult()) else None) return res } else async{return None} PostAndAsyncReply # This member uses the same functionality as PostAndTryAsyncReply, creating a message using the AsyncReplyChannel. The main difference is that an asynchronous workflow is created that wraps a call to PostAndTryAsyncReply if the timeout is specified.\nmember this.PostAndAsyncReply( replyChannelMsg, ?timeout) = let timeout = defaultArg timeout defaultTimeout match timeout with | Threading.Timeout.Infinite -\u0026gt; let resCell = AsyncResultCell\u0026lt;_\u0026gt;() let msg = replyChannelMsg (AsyncReplyChannel\u0026lt;_\u0026gt;(fun reply -\u0026gt; resCell.RegisterResult(reply) )) let posted = incomingMessages.Post(msg) if posted then resCell.AsyncWaitResult else raise (InvalidOperationException(\u0026#34;Incoming message buffer full.\u0026#34;)) | _ -\u0026gt; let asyncReply = this.PostAndTryAsyncReply(replyChannelMsg, timeout=timeout) async { let! res = asyncReply match res with | None -\u0026gt; return! raise (TimeoutException(\u0026#34;PostAndAsyncReply TimedOut\u0026#34;)) | Some res -\u0026gt; return res } Static Start # The static Start function is used as a way to construct and start the agent than using the constructor and then calling the Start function. This is really just a simple short cut for this common use case.\nstatic member Start(initial, ?cancellationToken) = let dfa = DataflowAgent\u0026lt;\u0026#39;Msg\u0026gt;(initial, ?cancellationToken = cancellationToken) dfa.Start() dfa Until next time\u0026hellip;\n","date":"January 30, 2012","externalUrl":null,"permalink":"/programming/2012-01-24-fsharp-dataflow-agents-ii/","section":"Blog","summary":"Right, no messing about this time, straight to the code.\nConstruction # This is pretty straight forward and I don’t want to detract from the important bits of this post, the only thing of note is the cancellationToken which is initialized to a default value using the defaultArg function if the optional parameter cancellationToken is not supplied. The TDF construct that we to use to perform most of the hard work is incomingMessages which is a BufferBlock\u003c'Msg\u003e.\n","title":"F# Dataflow Agents Part II","type":"programming"},{"content":"This is going to be a new series on using TPL Dataflow with F#. First a little bit of history and background.\nTPL Dataflows heritage and background # TPL Dataflow or (TDF) has been around for quite a while, it first surfaced more than a year ago as the successor to the Concurrency and Coordination Runtime (CCR) and with coming release of .Net 4.5 it will be part of the System.Threading.Tasks.Dataflow namespace. Elements of the now halted project Axum are also present within the design of TDF.\nConcurrency and Coordination Runtime (CCR) # CCR is a library that deals with asynchrony, concurrency, and coordination between blocks of asynchronous code so that the programmer doesn\u0026rsquo;t have to. All of the low level details of synchronization and error propagation are taken care of in a consistent fashion. CCR is still is included in Microsoft Robotics Studio where it is used extensively to exploit parallel hardware and deal with partial failure of systems.\nAxum # Axum was another interesting Microsoft research project, it also utilized the actor model embracing the principles of isolation, and message-passing. There was also extensive use symbolic operators as a terse short hand way to indicate operations between actors. For example \u0026lt;-- defined a way to pass a message to an actor. Theres was also a similarity to CCR as Axum used the concepts of Ports and channels in a similar way. It was a very interesting project and it was a shame it was put on hold.\nTPL Dataflow (TDF) # TDF builds on CCR and Axum, consolidating and refine to produce a more friendly fluent interface, much in the same vain as Language-Integrated Query (LINQ) and Reactive Extensions (RX).\nTDF is built around a number of different blocks which can be combined or linked together. There are three different categories of blocks are as follows:\nBuffering Blocks # Buffering blocks simply buffer data in various ways before passing the data on to another block.\nBufferBlock\u0026lt;\u0026lsquo;T\u0026gt; - The BufferBlock act as a first-in-first-out (FIFO) queue, buffering each input. BroadcastBlock\u0026lt;\u0026lsquo;T\u0026gt; - The BroadcastBlock linking to multiple targets copying the data to each of the connected blocks. WriteOnceBlock\u0026lt;\u0026lsquo;T\u0026gt; - The WriteOnceBlock acts like an immutable target, after an item first item is passed to it, it effectively becomes read only. Executor Blocks # The executor blocks run user supplied code in the form of a lambda expressions or a Task\u0026lt;'T\u0026gt;.\nActionBlock\u0026lt;\u0026lsquo;TInput\u0026gt; - The ActionBlock acts like the Action\u0026lt;'T\u0026gt; delegate performing an action on each datum posted to it. TransformBlock\u0026lt;\u0026lsquo;TInput,\u0026lsquo;TOutput\u0026gt; - The TransformBlock acts just like the ActionBlock except that the action performed can have an output, this output is buffered and behaves just like a BufferBlock. TransformManyBlock\u0026lt;\u0026lsquo;TInput,\u0026lsquo;TOutput\u0026gt; - The TransformManyBlock is just like a TransformBlock except that is can produce more than one output for a given datum. Joining Blocks # The Joining Blocks Combining or join data together in different ways.\nBatchBlock\u0026lt;\u0026lsquo;T\u0026gt; - The BatchBlock Combines multiple single items together, the items are represented by arrays of elements. The items are grouped together is batches and then passed on to another block. JoinBlock\u0026lt;\u0026lsquo;T1,\u0026lsquo;T2,…\u0026gt; - The JoinBlock acts as a form of Enumerable.Zip\u0026lt;'T1,'T2,'TResult\u0026gt; except the zip operation is performed on the items in the source array. BatchedJoinBlock\u0026lt;\u0026lsquo;T1,\u0026lsquo;T2,…\u0026gt; This block as the name suggests simply aggregates the JoinBlock and the BatchBlock together. Thats an ultra high level tour thats only just scratches the surface. I recommend you check out the Introduction to TPL Dataflow document to read up on the details. Theres a few more resources in the DevLabs area that you might find useful. Hopefully this series should also shed a bit more light on TDF as we go along\u0026hellip;\nF# Asynchronous Workflows and Agents # So where does that leave us in F#?\nIn F# we have Asynchronous Workflows and agents and they help immensely in the concurrency and message passing, but that doest mean that we cant take advantage of the new features and refinements much in the same way as we can use Asynchronous Workflows to take advantage of Tasks.\nThis post is going to be centered around F# agents but with a twist. First of all are going to be reimplementing a MailboxProcessor using TDF for the underlying processing. This will allow us to to use all of our existing agent code and examples and also stay within the F# agent paradigm. Following this approach we could also make use of the DataflowBlockOptions type, it has some interesting properties which we will look at in future posts:\nTaskScheduler CancellationToken MaxMessagesPerTask BoundedCapacity Implementation # In this post we are going replicate the MailboxProcessor, we will be using Tomas Petricek\u0026rsquo;s caching agent example from FSSnip). I have made a couple of modification to Tomas\u0026rsquo;s code.\nI replaced the Dictionary type with a ConcurrentDictionary so that the caching agent could be called multiple times successively without the dictionary throwing an exception due to it already containing a key from a previous cached result. I also changed the example code so that it requests cached HTML from the caching agent ten times with a 400ms interval in between each.\nmodule TplAgents open System open System.Collections.Generic open System.Collections.Concurrent open FsDataflow open System.Net open Microsoft.FSharp.Control.WebExtensions type CachingMessage = | Add of string * string | Get of string * AsyncReplyChannel\u0026lt;option\u0026lt;string\u0026gt;\u0026gt; | Clear let caching = DataflowAgent.Start(fun agent -\u0026gt; async { let table = ConcurrentDictionary\u0026lt;string, string\u0026gt;() while true do let! msg = agent.Receive() match msg with | Add(url, html) -\u0026gt; // Add downloaded page to the cache table.AddOrUpdate(url, html, fun k v -\u0026gt; html) |\u0026gt; ignore | Get(url, repl) -\u0026gt; // Get a page from the cache - returns // None if the value isn\u0026#39;t in the cache if table.ContainsKey(url) then repl.Reply(Some table.[url]) else repl.Reply(None) | Clear -\u0026gt; table.Clear() }) /// Prints information about the specified web site using cache let printInfo url = async { // Try to get the cached HTML from the caching agent let! htmlOpt = caching.PostAndAsyncReply(fun ch -\u0026gt; Get(url, ch)) match htmlOpt with | None -\u0026gt; // New url - download it and add it to the cache use wc = new WebClient() let! text = wc.AsyncDownloadString(Uri(url)) caching.Post(Add(url, text)) Console.WriteLine( sprintf \u0026#34;Download: %s (%d)\u0026#34; url text.Length) | Some html -\u0026gt; // The url was downloaded earlier Console.WriteLine( sprintf \u0026#34;Cached: %s (%d)\u0026#34; url html.Length) } let printfuncpro = printInfo \u0026#34;http://functional-programming.net\u0026#34; // Print information about a web site - // Run this repeatedly to use cached value for i in 1 .. 10 do printfuncpro |\u0026gt; Async.Start Async.RunSynchronously \u0026lt;| Async.Sleep 400 // Clear the cache - \u0026#39;printInfo\u0026#39; will need to // download data from the web site again Console.WriteLine(sprintf \u0026#34;Clearing the cache\u0026#34;) caching.Post(Clear) printfuncpro |\u0026gt; Async.Start Console.ReadKey() |\u0026gt; ignore Looking at the implementation above you can see that we need to implement the following members:\nStart:unit -\u0026gt; unit Receive:?int -\u0026gt; Async\u0026lt;'Msg\u0026gt; Post:'Msg -\u0026gt; unit PostAndTryAsyncReply:(AsyncReplyChannel\u0026lt;'Reply\u0026gt; -\u0026gt; 'Msg) * ?int -\u0026gt; Async\u0026lt;'Reply option\u0026gt; PostAndAsyncReply:(AsyncReplyChannel\u0026lt;'Reply\u0026gt; -\u0026gt; 'Msg) * int option -\u0026gt; Async\u0026lt;'Reply\u0026gt; static member Start:(MailboxProcessor\u0026lt;'Msg\u0026gt; -\u0026gt; Async\u0026lt;unit\u0026gt;) * ?CancellationToken -\u0026gt; MailboxProcessor\u0026lt;'Msg\u0026gt; These are the only members we need to complete the caching agent example, I didn\u0026rsquo;t want bamboozle everyone with an explosion of code from the onset so the remaining members will be implemented as and when we need them. When we have implemented all the members from MailboxProcessor Ill post the full source on my GitHub account.\nThe following members will be outstanding but it should be fairly trivial to implement them once we have completed the code here.\nPostAndReply:(AsyncReplyChannel\u0026lt;'Reply\u0026gt; -\u0026gt; 'Msg) * int option -\u0026gt; 'Reply Scan:('Msg -\u0026gt; Async\u0026lt;'T\u0026gt; option) * ?int -\u0026gt; Async\u0026lt;'T\u0026gt; TryPostAndReply:(AsyncReplyChannel\u0026lt;'Reply\u0026gt; -\u0026gt; 'Msg) * ?int -\u0026gt; 'Reply option TryReceive:?int -\u0026gt; Async\u0026lt;'Msg option\u0026gt; TryScan:('Msg -\u0026gt; Async\u0026lt;'T\u0026gt; option) * ?int -\u0026gt; Async\u0026lt;'T option\u0026gt; CurrentQueueLength:int DefaultTimeout:int with get, set So here we go, this is the Dataflow implementation of the MailboxProcessor:\nmodule FsDataflow open System open System.Threading open System.Threading.Tasks open System.Threading.Tasks.Dataflow open System.Collections.Concurrent type DataflowAgent\u0026lt;\u0026#39;Msg\u0026gt;(initial, ?cancellationToken) = let cancellationToken = defaultArg cancellationToken Async.DefaultCancellationToken let mutable started = false let errorEvent = new Event\u0026lt;System.Exception\u0026gt;() let incomingMessages = new BufferBlock\u0026lt;\u0026#39;Msg\u0026gt;() let mutable defaultTimeout = Timeout.Infinite [\u0026lt;CLIEvent\u0026gt;] member this.Error = errorEvent.Publish member this.Start() = if started then raise (new InvalidOperationException(\u0026#34;Already Started.\u0026#34;)) else started \u0026lt;- true let comp = async { try do! initial this with error -\u0026gt; errorEvent.Trigger error } Async.Start(computation = comp, cancellationToken = cancellationToken) member this.Receive(?timeout) = Async.AwaitTask \u0026lt;| incomingMessages.ReceiveAsync() member this.Post(item) = let posted = incomingMessages.Post(item) if not posted then raise (InvalidOperationException(\u0026#34;Incoming message buffer full.\u0026#34;)) member this.PostAndTryAsyncReply(replyChannelMsg, ?timeout) = let timeout = defaultArg timeout defaultTimeout let resultCell = AsyncResultCell\u0026lt;_\u0026gt;() let msg = replyChannelMsg(AsyncReplyChannel\u0026lt;_\u0026gt;(fun reply -\u0026gt; resultCell.RegisterResult(reply))) let posted = incomingMessages.Post(msg) if posted then match timeout with | Threading.Timeout.Infinite -\u0026gt; async { let! result = resultCell.AsyncWaitResult return Some(result) } | _ -\u0026gt; async { let! ok = resultCell.GetWaitHandle(timeout) let res = (if ok then Some(resultCell.GrabResult()) else None) return res } else async{return None} member this.PostAndAsyncReply( replyChannelMsg, ?timeout) = let timeout = defaultArg timeout defaultTimeout match timeout with | Threading.Timeout.Infinite -\u0026gt; let resCell = AsyncResultCell\u0026lt;_\u0026gt;() let msg = replyChannelMsg (AsyncReplyChannel\u0026lt;_\u0026gt;(fun reply -\u0026gt; resCell.RegisterResult(reply) )) let posted = incomingMessages.Post(msg) if posted then resCell.AsyncWaitResult else raise (InvalidOperationException(\u0026#34;Incoming message buffer full.\u0026#34;)) | _ -\u0026gt; let asyncReply = this.PostAndTryAsyncReply(replyChannelMsg, timeout=timeout) async { let! res = asyncReply match res with | None -\u0026gt; return! raise (TimeoutException(\u0026#34;PostAndAsyncReply TimedOut\u0026#34;)) | Some res -\u0026gt; return res } static member Start(initial, ?cancellationToken) = let dfa = DataflowAgent\u0026lt;\u0026#39;Msg\u0026gt;(initial, ?cancellationToken = cancellationToken) dfa.Start() dfa The crux of the implementation from TDF\u0026rsquo;s point of view is the use of the BufferBlock.\nThis is one of the most fundamental blocks within TDF. Its the equivalent of the Port\u0026lt;'T\u0026gt; type from CCR and the Mailbox type from F# which is used internally within the MailboxProcessor. As mentioned abouve the BufferBlock type is a first-in-first-out (FIFO) buffer and is responsible for buffering any data that is Posted to it.\nOK, I\u0026rsquo;m going to leave it at that for now while you digest the code presented here.\nIn part II I will be drilling into the detail on whats going on internally and also describing more of the TDF model, so tune in soon for Part II.\nUntil next time\u0026hellip;\n","date":"January 22, 2012","externalUrl":null,"permalink":"/programming/2012-01-22-fsharp-dataflow-agents-i/","section":"Blog","summary":"This is going to be a new series on using TPL Dataflow with F#. First a little bit of history and background.\nTPL Dataflows heritage and background # TPL Dataflow or (TDF) has been around for quite a while, it first surfaced more than a year ago as the successor to the Concurrency and Coordination Runtime (CCR) and with coming release of .Net 4.5 it will be part of the System.Threading.Tasks.Dataflow namespace. Elements of the now halted project Axum are also present within the design of TDF.\n","title":"F# Dataflow Agents Part I","type":"programming"},{"content":"Due to popular demand\u0026hellip; well, I had a couple of requests anyway :-) Heres a post inspired by my recent encounters profiling some of the code in Fracture-IO. I have recently been profiling the code in fracture to remove any so called low hanging fruits. During this time I also noticed an increase in memory allocation. I remembered I had recently been experimenting in a branch using pipelets as a buffer between the send and receive stages in the Http Server, so I set up a simple test to see if pipelets were contributing to the memory allocation issues I was seeing. Here\u0026rsquo;s the simple iteration test code I used for the memory profiling:\nopen System open System.Diagnostics open System.Threading open Fracture.Pipelets let reverse (s:string) = String(s |\u0026gt; Seq.toArray |\u0026gt; Array.rev) let oneToSingleton a b f= let result = b |\u0026gt; f result |\u0026gt; Seq.singleton /// Total number to run through test cycle let number = 100 /// To Record when we are done let counter = ref 0 let sw = new Stopwatch() let countThis (a:String) = do Interlocked.Increment(counter) |\u0026gt; ignore if !counter % number = 0 then sw.Stop() printfn \u0026#34;Execution time: %A\u0026#34; sw.Elapsed.TotalMilliseconds printfn \u0026#34;Items input: %d\u0026#34; number printfn \u0026#34;Time per item: %A ms (Elapsed Time / Number of items)\u0026#34; (TimeSpan.FromTicks(sw.Elapsed.Ticks / int64 number).TotalMilliseconds) printfn \u0026#34;Press any key to repeat, press \u0026#39;q\u0026#39; to exit.\u0026#34; sw.Reset() counter |\u0026gt; Seq.singleton let OneToSeqRev a b = oneToSingleton a b reverse let generateCircularSeq (s) = let rec next () = seq { for element in s do yield element yield! next() } next() let stage1 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage1\u0026#34;, OneToSeqRev \u0026#34;1\u0026#34;, Routers.roundRobin, number, -1) let stage2 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage2\u0026#34;, OneToSeqRev \u0026#34;2\u0026#34;, Routers.basicRouter, number, -1) let stage3 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage3\u0026#34;, OneToSeqRev \u0026#34;3\u0026#34;, Routers.basicRouter, number, -1) let stage4 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage4\u0026#34;, OneToSeqRev \u0026#34;4\u0026#34;, Routers.basicRouter, number, -1) let stage5 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage5\u0026#34;, OneToSeqRev \u0026#34;5\u0026#34;, Routers.basicRouter, number, -1) let stage6 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage6\u0026#34;, OneToSeqRev \u0026#34;6\u0026#34;, Routers.basicRouter, number, -1) let stage7 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage7\u0026#34;, OneToSeqRev \u0026#34;7\u0026#34;, Routers.basicRouter, number, -1) let stage8 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage8\u0026#34;, OneToSeqRev \u0026#34;8\u0026#34;, Routers.basicRouter, number, -1) let stage9 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage9\u0026#34;, OneToSeqRev \u0026#34;9\u0026#34;, Routers.basicRouter, number, -1) let stage10 = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Stage10\u0026#34;, OneToSeqRev \u0026#34;10\u0026#34;, Routers.basicRouter, number, -1) let final = new Pipelet\u0026lt;_,_\u0026gt;(\u0026#34;Final\u0026#34;, countThis, Routers.basicRouter, number, -1) let manyStages = [stage2;stage3;stage4;stage5;stage6;stage7;stage8;stage9;stage10] oneToMany stage1 manyStages manyToOne manyStages final System.AppDomain.CurrentDomain.UnhandledException |\u0026gt; Observable.add (fun x -\u0026gt; printfn \u0026#34;%A\u0026#34; (x.ExceptionObject :?\u0026gt; Exception); Console.ReadKey() |\u0026gt; ignore) let circ = [\u0026#34;John\u0026#34;; \u0026#34;Paul\u0026#34;; \u0026#34;George\u0026#34;; \u0026#34;Ringo\u0026#34;; \u0026#34;Nord\u0026#34;; \u0026#34;Bert\u0026#34;] |\u0026gt; generateCircularSeq let startoperations() = sw.Start() for str in circ |\u0026gt; Seq.take number do str --\u0026gt; stage1 printfn \u0026#34;Insert complete waiting for operation to complete.\u0026#34; printfn \u0026#34;Press any key to process %i items\u0026#34; number while not (Console.ReadKey().Key = ConsoleKey.Q) do startoperations() Using process explorer from Mark Russinovich I watched the allocated memory grow as the iterations progressed:\nTheres definitely something leaking in there! So what can we do to find this? Simple, we use a memory profiler. There are several really good memory profilers out there. I have listed some of the best ones below:\nSciTech memory profiler RedGates ANTS Memory Profiler JetBrains dotTrace YourKit Profiler for .NET To demonstrate finding the leak I will be using [RedGates ANTS MemoryProfiler](http://www.red-gate.com/products/dotnet-development/ants-memory- profiler/). First of all we launch the profiler and set it up to profile the application, this is just a simple case of browsing to the release folder and picking the application so I won\u0026rsquo;t bore with those trivial details here. Now that the application is running we hit any key which caused the test application to post 100 operations into the pipeline. We want to create a baseline snapshot of the memory allocation so we can see where our leak is. To do this click Take Memory Snapshot at the top right of the screen. Next we hit any key again in the test application, again causing it to post another 100 operations into the pipeline. Now we click Take Memory Snapshot again. Now we have a snapshot of the difference between the two operations. The summery screen is shown below:\nFrom this screen you can see that there is 51.56KB of new memory allocated since the last snapshot, and you can see some nice piecharts showing the various allocations in G1, G2 etc. On the right hand side of the pie chart you can see that the largest classes are: object[], AsyncParamsAux, Pipelets+loop@37-7\u0026lt;Unit, string,string\u0026gt;, and AsyncParams.\nNow if we click on Class List button we can investigate these further, heres the Class List:\nHere things start to get interesting. If you click on the instance Diff (+/-) column you can sort the list of classed by the differences to the last snapshot.\nNow looking at the results we have:\n300 more instances of AsyncBuilderImpl, AsyncParamsArgs, and AsyncParams 200 more instances of Pipelets+loop@37-7\u0026lt;Unit, string, string\u0026gt; 100 more instances of Pipelets+loop@37-7\u0026lt;Unit, string, FSharpRef\u0026gt; Is it a coincidence that we just pushed 100 operations through the pipeline? I think not!\nNow that we have a target for further inspection we can highlight the row for the function Pipelets+loop@37-7\u0026lt;Unit, string, FSharpRef\u0026raquo; and then click on the icon that has three little blue boxes on it. This will take us to the instance List as shown below:\nI have sorted the instance list by the distance from the GC Root, you can see there is a strange pattern emerging, the GC root distant increase by three each time. Now lets look at the Instance Retention graph for the first one with a GC Root distance of 9, this is the icon on the right hand side of the function name, it looks like a few rectangles joined up with a line:\nThe Pipelets+loop function is linked from the mailbox processor shown at the top of the graph and flows into the Async infrastructure, and finally to the loop function at the bottom.\nLets look at the next one, this has a GC Root distance of 12:\nIf you look carefully there is another pattern here, the field references args, aux@, econt@ are repeated in the red boxes. The functions look to be quite similar too. Lets look at the next one GC Root Distance of 15:\nLooking at this we have a definite repeat of the functions and arguments, if we look down to GC Root at a depth of 60 we get this:\nSo whats happening here is that there is a continuation that has been built around the asynchronous calls that gets bigger and bigger on each iteration.\nNow that we have identified the leak, lets look at the code and see whats going on. That would be the loop function in Pipelets:\nlet mailbox = MailboxProcessor.Start(fun inbox -\u0026gt; let rec loop routes = async { let! msg = inbox.Receive() match msg with | Payload(data) -\u0026gt; ss.Release() |\u0026gt; ignore try data |\u0026gt; transform |\u0026gt; router \u0026lt;| routes return! loop routes with //force loop resume on error | ex -\u0026gt; errors ex return! loop routes | Attach(stage) -\u0026gt; return! loop (stage::routes) | Detach(stage) -\u0026gt; return! loop (List.filter (fun x -\u0026gt; x \u0026lt;\u0026gt; stage) routes) } loop []) Have a look at lines 9 and 12. Can you guess whats wrong?\nWell, to quote the F# Teams blog:\nOn the .NET platform, there are limitations on where tail calls may occur. One restriction is that tail calls cannot be performed in try-catch or try- finally blocks (neither in the body of the try nor in the catch or finally handlers).\nIt goes on further to discuss another subtle issue with use bindings:\nuse bindings implicitly generate a try-finally around the code that follows them to ensure that the Dispose method is called on the bound value. This means that no calls following a use binding will be tail calls.\nSo all we have to do change the way the try catch block is formulated in that section. The most idiomatic way of dealing with this is to use the Async.Catch function which would result in code something like the following:\nlet mailbox = MailboxProcessor.Start(fun inbox -\u0026gt; let rec loop routes = async { let! msg = inbox.Receive() match msg with | Payload(data) -\u0026gt; ss.Release() |\u0026gt; ignore let result = async{data |\u0026gt; transform |\u0026gt; router \u0026lt;| routes} |\u0026gt; Async.Catch |\u0026gt; Async.RunSynchronously match result with | Choice1Of2() -\u0026gt; () | Choice2Of2 exn -\u0026gt; errors exn return! loop routes | Attach(stage) -\u0026gt; return! loop (stage::routes) | Detach(stage) -\u0026gt; return! loop (List.filter (fun x -\u0026gt; x \u0026lt;\u0026gt; stage) routes) } loop []) Alternatively you could move the entire try with section out to a more local section thats not in the recursive async loop construct:\nlet computeAndRoute data routes = try data |\u0026gt; transform |\u0026gt; router \u0026lt;| routes Choice1Of2() with | ex -\u0026gt; Choice2Of2 ex let mailbox = MailboxProcessor.Start(fun inbox -\u0026gt; let rec loop routes = async { let! msg = inbox.Receive() match msg with | Payload(data) -\u0026gt; ss.Release() |\u0026gt; ignore match computeAndRoute data routes with | Choice2Of2 exn -\u0026gt; errors exn | _ -\u0026gt; () return! loop routes | Attach(stage) -\u0026gt; return! loop (stage::routes) | Detach(stage) -\u0026gt; return! loop (List.filter (fun x -\u0026gt; x \u0026lt;\u0026gt; stage) routes)} loop []) Anyway I hope that sheds a bit of light on how to spot where memory leaks are stemming from, and also some of the little known and often forgotten caveats with tail recursion.\nUntil next time\u0026hellip;\nEDIT: Just to make things a little bit clearer. The memory leak here is caused by the async block being transformed into chains of continuation passing-style functions, and due to tail call elimination not being possible inside of the try catch blocks, the continuation grows and grows during each recursion.\n","date":"December 11, 2011","externalUrl":null,"permalink":"/programming/2011-12-11-fixing-a-hole/","section":"Blog","summary":"Due to popular demand… well, I had a couple of requests anyway :-) Heres a post inspired by my recent encounters profiling some of the code in Fracture-IO. I have recently been profiling the code in fracture to remove any so called low hanging fruits. During this time I also noticed an increase in memory allocation. I remembered I had recently been experimenting in a branch using pipelets as a buffer between the send and receive stages in the Http Server, so I set up a simple test to see if pipelets were contributing to the memory allocation issues I was seeing. Here’s the simple iteration test code I used for the memory profiling:\n","title":"Fixing a hole...","type":"programming"},{"content":" SOLID and its relevance to F# # There has been an increasing amount of exposure for F# and functional programming lately. If you come from an object-orientated background a change in mindset is required when working with functional programming, there is a lot of misinformation on functional languages and their relationship with object-orientated design. In this post we run quickly through SOLID to see if these object-orientated principles apply to F#, and if so, how.\nThis post assumes you are familiar with SOLID principles, if not here is a [link](http://en.wikipedia.org/wiki/SOLID_(object-oriented_design). Lets take a quick overview of what SOLID stands for:\nSingle responsibility principle # This is the notion that an object should have only a single responsibility.\nAn object-orientated program consists of layers of abstract classes with less abstract classes layered on top of ones that are more abstract. Functional programming is similar, although abstractions are used throughout the design and are composed into a final solution. Programming in F# naturally forms small succinct functions which should have a single purpose, so the single responsibility rule holds strong here.\nOpen closed principle # The notion that “software entities should be open for extension, but closed for modification”\nThis comes down to building abstractions via inheritance, and behavior changes through polymorphism. Early on a decision must be made on which parts will change, and which will be fixed. The open closed principal is geared towards languages where inheritance is a core concept, inheritance and polymorphism are not strongly used in F#, so this principle is very weak here. Composition and Type augmentation are the core methods for extension in F#.\nLiskov substitution principle # Liskov substitution states “objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program”.\nThis principle is all about inheritance - derived types preserving the specification of base types. Functional languages like F# do not always use inheritance, it is used quite rarely and only in certain situations. The idioms in the language like referential transparency lean strongly towards enforcing this principle. Functional languages also have a heritage in mathematics and algebraic reasoning so referential transparency is key in this respect.\nInterface segregation principle # The notion that “many client specific interfaces are better than one general purpose interface.”\nIf you are violating single responsibility then your interface will probably be bloated too with unnecessary properties and methods. The same rule applies to F#, keep your interfaces modular and keep indifferent concepts separated.\nDependency inversion principle # The notion that one should “Depend upon Abstractions. Do not depend upon concretions.”\nHigh-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend upon details. Details should depend upon abstractions. Dependency inversion is often solved with concepts such as dependency injection; common techniques for this involves things like constructor injection and interface injection. Inversion of control is often solved with a factory pattern or a service locator. From a functional point of view, these containers and injection concepts can be solved with a simple higher order function, or hole-in-the-middle type pattern which are built right into the language. Summary # So we can assume that in the functional paradigm:\nTypes should have only a single responsibility. The open closed principle doesn\u0026rsquo;t apply in the majority of cases unless we are using an object-oriented approach. With functional programming we favour functional composition and type augmentation. The same can be said of the Liskov substitute principle, pure functions uphold this tenant via immutability and referential transparency. Interface segregation also applies and idiomatic F# produces small concise functions with modular interfaces with a separation of concerns. Dependency inversion is not a relevant issue for F# as the language itself supports higher-order-functions to encapsulate this concept. So only the \u0026lsquo;S\u0026rsquo; and the \u0026lsquo;I\u0026rsquo; parts are relevant in functional programming. The other tenants are not fully representative as they stand. We could probably explore the tenants of good functional design and form a quirky acronym in the process, but Ill leave that for another time. Until next time\u0026hellip;\n","date":"August 22, 2011","externalUrl":null,"permalink":"/programming/2011-08-22-fsharp_solid/","section":"Blog","summary":"SOLID and its relevance to F# # There has been an increasing amount of exposure for F# and functional programming lately. If you come from an object-orientated background a change in mindset is required when working with functional programming, there is a lot of misinformation on functional languages and their relationship with object-orientated design. In this post we run quickly through SOLID to see if these object-orientated principles apply to F#, and if so, how.\n","title":"F# and Design principles - SOLID","type":"programming"},{"content":"I recently wrote an article for Developer Fusion on the changes in mindset required when moving from C# to F#.\nThe article has proved to be more more popular than I envisaged. I think a lot of .Net developers are interested in F# but are unsure on the path to take when trying to accomplish this. For me it was almost a leap of faith, I saw the potential benefits and just jumped right in.\nI had to overcome numerous obstacles along the way before I become comfortable within the language. I had question like:\nHow do design patterns and principles fit in. How do I structure my applications. How can I work seamlessly with other libraries in the .Net ecosphere. I will try and answer some of these question over the coming weeks as well as introducing some new topics. If anyone has any comments on the article or suggestions on future content please leave them below, and I will try to work them into future posts.\nYou can find the article here.\nUntil next time\u0026hellip;\n","date":"July 16, 2011","externalUrl":null,"permalink":"/programming/2011-07-16-from-csharp-to-fsharp-a-developers-perspective/","section":"Blog","summary":"I recently wrote an article for Developer Fusion on the changes in mindset required when moving from C# to F#.\nThe article has proved to be more more popular than I envisaged. I think a lot of .Net developers are interested in F# but are unsure on the path to take when trying to accomplish this. For me it was almost a leap of faith, I saw the potential benefits and just jumped right in.\n","title":"From C# to F#: A Developer's Perspective","type":"programming"},{"content":"One of the areas that I am very interested in is agents and I have been doing quite a lot of work in this area lately.\nAgents can be used for a multitude of different purposes ranging from: isolated message passing, object caching, finite state machines, web crawling, and even reactive user interfaces. One of the ideas that I have been looking into lately is agent based scheduling.\nSchedulerAgent # A simple Agent based scheduler:\nmodule AgentUtilities open System open System.Threading //Agent alias for MailboxProcessor type Agent\u0026lt;\u0026#39;T\u0026gt; = MailboxProcessor\u0026lt;\u0026#39;T\u0026gt; /// Two types of Schedule messages that can be sent type ScheduleMessage\u0026lt;\u0026#39;a\u0026gt; = | Schedule of (\u0026#39;a -\u0026gt; unit) * \u0026#39;a * TimeSpan * TimeSpan * CancellationTokenSource AsyncReplyChannel | ScheduleOnce of (\u0026#39;a -\u0026gt; unit) * \u0026#39;a * TimeSpan * CancellationTokenSource AsyncReplyChannel /// An Agent based scheduler type SchedulerAgent\u0026lt;\u0026#39;a\u0026gt;()= let scheduleOnce delay msg receiver (cts: CancellationTokenSource)= async { do! Async.Sleep(delay) if (cts.IsCancellationRequested) then cts.Dispose() else msg |\u0026gt; receiver } let scheduleMany initialDelay msg receiver delayBetween cts= let rec loop time (cts: CancellationTokenSource) = async { do! Async.Sleep(time) if (cts.IsCancellationRequested) then cts.Dispose() else msg |\u0026gt; receiver return! loop delayBetween cts} loop initialDelay cts let scheduler = Agent.Start(fun inbox -\u0026gt; let rec loop() = async { let! msg = inbox.Receive() let cs = new CancellationTokenSource() match msg with | Schedule(receiver, msg:\u0026#39;a, initialDelay, delayBetween, replyChan) -\u0026gt; Async.StartImmediate(scheduleMany (int initialDelay.TotalMilliseconds) msg receiver (int delayBetween.TotalMilliseconds) cs ) replyChan.Reply(cs) return! loop() | ScheduleOnce(receiver, msg:\u0026#39;a, delay, replyChan) -\u0026gt; Async.StartImmediate(scheduleOnce (int delay.TotalMilliseconds) msg receiver cs) replyChan.Reply(cs) return! loop() } loop()) ///Schedules a message to be sent to the receiver after the initialDelay. /// If delaybetween is specified then the message is sent reoccuringly at the delaybetween interval. member this.Schedule(receiver, msg, initialDelay, ?delayBetween) = let buildMessage replyChan = match delayBetween with | Some(x) -\u0026gt; Schedule(receiver,msg,initialDelay, x, replyChan) | _ -\u0026gt; ScheduleOnce(receiver,msg,initialDelay, replyChan) scheduler.PostAndReply (fun replyChan -\u0026gt; replyChan |\u0026gt; buildMessage) The structure of the SchedulerAgent broken down into sections below:\nScheduleMessage # Lines 9-11 (type ScheduleMessage\u0026lt;'a\u0026gt; =) show the definition of ScheduleMessage. This is a discriminated union of two different types of Schedule message.\nScheduleOnce # ScheduleOnce has four parameters:\nA function which is called at the schedule time (\u0026lsquo;a -\u0026gt; unit). The message that is sent at the schedules time (\u0026lsquo;a). A TimeSpan which is the length of time to wait before triggering the schedule. An AsyncReplyChannel(CancellationTokenSource AsyncReplyChannel). This is used to return a CancellationTokenSource which can be used to cancel the Schedule. Schedule # Schedule has five parameters which are as follows:\nA function which is called at the schedule time (\u0026lsquo;a -\u0026gt; unit). The message that is sent at the schedules time (\u0026lsquo;a). A TimeSpan which is the initial length of time to wait before first triggering the schedule function. A TimeSpan which is used as an interval between each subsequent triggering of the schedule function. An AsyncReplyChannel(CancellationTokenSource AsyncReplyChannel). This is used to return a CancellationTokenSource which can be used to cancel the Schedule. SchedulerAgent # scheduleOnce # Lines 16-20 define an async workflow, which asynchronously sleeps for the specified time before checking that the schedule hasn\u0026rsquo;t been cancelled before finally calling the schedule function.\nscheduleMany # Lines 22-29 define a recursive async workflow, which asynchronously sleeps for the specified interval (3rd Parameter) before checking the schedule hasn\u0026rsquo;t been cancelled before finally calling the schedule function. The loop function is then called passing in the second TimeSpan interval (4th Parameter).\nscheduler # This is the main processing loop for the agent. A recursive loop function is declared on line 32. On line 33 the agent waits for a message to arrive. Once a message arrives a CancellationTokenSource is created on line 36 which can be used to cancel an already scheduled message. Pattern matching is used on line 35 to find the type of message that has been received. The first pattern matching block on lines 36-43 matches the Schedule message. The parameters from the Schedule message are passed into the scheduleMany function. This is then invoked asynchronously via the Async.StartImmediate function. The CancellationTokenSource is now returned to the caller on line 43. This allows the caller to cancel an already running schedule. Finally the recursive loop function is called on line 44. The second pattern matching block on lines 45-52 is much the same passing the parameters from the ScheduleOnce message into the scheduleOnce function, again this is invoked via the Async.StartImmediate function. Like the Schedule message the CancellationTokenSource returned on line 51 and the recursive loop function is called on line 52.\nThe agent is then started on line 51 by calling the loop function for the first time.\nMembers # The SchedulerAgent has only a single member Schedule. This member function takes three parameters and an optional parameter delayBetween. A function called buildMessage on line 59 uses the optional parameter with pattern matching to determine whether a ScheduleOnce or a Schedule message is created. The agent is posted the correct message type on line 63 using the synchronous call scheduler.PostAndReply. We use a synchronous call to return the cancellationTokenSource immediately, and this can be used to cancel a running schedule.\nSample Application # Shows a test harness that creates and uses a simple string based message scheduler:\nopen AgentUtilities open System open System.Threading let scheduler = SchedulerAgent\u0026lt;_\u0026gt;() let printer message = printfn \u0026#34;%s: %s\u0026#34; (DateTime.Now.TimeOfDay.ToString()) message let singlecancel = scheduler.Schedule(printer, \u0026#34;Hello from the scheduler\u0026#34;, TimeSpan(0,0,0,5)) let multicancel = scheduler.Schedule( printer, \u0026#34;Hello from the multi scheduler\u0026#34;, TimeSpan(0,0,0,5), TimeSpan(0,0,0,0,500)) printfn \u0026#34;Press any key to cancel.\u0026#34; Console.ReadKey() |\u0026gt; ignore //Cancel the multi scheduler multicancel.Cancel() printfn \u0026#34;Cancelled, press any key to exit.\u0026#34; Console.ReadKey() |\u0026gt; ignore I hope this gives you a feel for what you can do with agent based scheduling. The library here could be expanded further in several ways. You could replace the fixed message with a message generator function or even an agent based message generator. If the schedule function was abstracted somewhat it could be made to accept an agent as the receiver.\nOne of the key areas I am looking at is building a distributed agent library that would allow an agent to communicate over network layers transparently. A scheduler agent would be even more powerful in this environment. I could envisage them used for a many different things in this environment: heart beat messages, performance sampling, diagnostics and testing.\nUntil next time\u0026hellip;\n","date":"July 3, 2011","externalUrl":null,"permalink":"/programming/2011-07-03-agent-based-scheduling/","section":"Blog","summary":"One of the areas that I am very interested in is agents and I have been doing quite a lot of work in this area lately.\nAgents can be used for a multitude of different purposes ranging from: isolated message passing, object caching, finite state machines, web crawling, and even reactive user interfaces. One of the ideas that I have been looking into lately is agent based scheduling.\n","title":"Agent based scheduling","type":"programming"},{"content":"Everyone knows F# agents are cool right? Well here\u0026rsquo;s yet another example of how versatile they can be\u0026hellip;\nThere was a series of posts last April by Stephen Toub from the pfxteam at Microsoft. I was reading through some of the posts again the other day and thought some of the ideas presented there would make interesting projects in F# to demonstrate the flexibility and succinctness of the language. I thought the ObjectPool example would make an interesting project in F# using agents aka MailboxProcessors. An ObjectPool is basically a pool of objects that have been pre-created so that you can grab one and use it, and then place it back in the pool when you\u0026rsquo;re finished. They are useful in situations where the cost of creating object from scratch is very high or you want to cut down on allocations in the garbage collector.\nFirst of all heres the C# code as it was presented in the Parallel Extensions download:\nusing System.Collections.Generic; using System.Diagnostics; namespace System.Collections.Concurrent { /// \u0026lt;summary\u0026gt;Provides a thread-safe object pool.\u0026lt;/summary\u0026gt; /// \u0026lt;typeparam name=\u0026#34;T\u0026#34;\u0026gt;Specifies the type of the elements stored in the pool.\u0026lt;/typeparam\u0026gt; [DebuggerDisplay(\u0026#34;Count={Count}\u0026#34;)] [DebuggerTypeProxy(typeof(IProducerConsumerCollection_DebugView\u0026lt;\u0026gt;))] public sealed class ObjectPool\u0026lt;T\u0026gt; : ProducerConsumerCollectionBase\u0026lt;T\u0026gt; { private readonly Func\u0026lt;T\u0026gt; _generator; /// \u0026lt;summary\u0026gt;Initializes an instance of the ObjectPool class.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;generator\u0026#34;\u0026gt;The function used to create items when no items exist in the pool.\u0026lt;/param\u0026gt; public ObjectPool(Func\u0026lt;T\u0026gt; generator) : this(generator, new ConcurrentQueue\u0026lt;T\u0026gt;()) { } /// \u0026lt;summary\u0026gt;Initializes an instance of the ObjectPool class.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;generator\u0026#34;\u0026gt;The function used to create items when no items exist in the pool.\u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;collection\u0026#34;\u0026gt;The collection used to store the elements of the pool.\u0026lt;/param\u0026gt; public ObjectPool(Func\u0026lt;T\u0026gt; generator, IProducerConsumerCollection\u0026lt;T\u0026gt; collection) : base(collection) { if (generator == null) throw new ArgumentNullException(\u0026#34;generator\u0026#34;); _generator = generator; } /// \u0026lt;summary\u0026gt;Adds the provided item into the pool.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;item\u0026#34;\u0026gt;The item to be added.\u0026lt;/param\u0026gt; public void PutObject(T item) { base.TryAdd(item); } /// \u0026lt;summary\u0026gt;Gets an item from the pool.\u0026lt;/summary\u0026gt; /// \u0026lt;returns\u0026gt;The removed or created item.\u0026lt;/returns\u0026gt; /// \u0026lt;remarks\u0026gt;If the pool is empty, a new item will be created and returned.\u0026lt;/remarks\u0026gt; public T GetObject() { T value; return base.TryTake(out value) ? value : _generator(); } /// \u0026lt;summary\u0026gt;Clears the object pool, returning all of the data that was in the pool.\u0026lt;/summary\u0026gt; /// \u0026lt;returns\u0026gt;An array containing all of the elements in the pool.\u0026lt;/returns\u0026gt; public T[] ToArrayAndClear() { var items = new List\u0026lt;T\u0026gt;(); T value; while (base.TryTake(out value)) items.Add(value); return items.ToArray(); } protected override bool TryAdd(T item) { PutObject(item); return true; } protected override bool TryTake(out T item) { item = GetObject(); return true; } } } There\u0026rsquo;s also a base class which looks like this:\n/// \u0026lt;summary\u0026gt; /// Provides a base implementation for producer-consumer collections that wrap other /// producer-consumer collections. /// \u0026lt;/summary\u0026gt; /// \u0026lt;typeparam name=\u0026#34;T\u0026#34;\u0026gt;Specifies the type of elements in the collection.\u0026lt;/typeparam\u0026gt; [Serializable] public abstract class ProducerConsumerCollectionBase\u0026lt;T\u0026gt; : IProducerConsumerCollection\u0026lt;T\u0026gt; { private readonly IProducerConsumerCollection\u0026lt;T\u0026gt; _contained; /// \u0026lt;summary\u0026gt;Initializes the ProducerConsumerCollectionBase instance.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;contained\u0026#34;\u0026gt;The collection to be wrapped by this instance.\u0026lt;/param\u0026gt; protected ProducerConsumerCollectionBase(IProducerConsumerCollection\u0026lt;T\u0026gt; contained) { if (contained == null) throw new ArgumentNullException(\u0026#34;contained\u0026#34;); _contained = contained; } /// \u0026lt;summary\u0026gt;Gets the contained collection.\u0026lt;/summary\u0026gt; protected IProducerConsumerCollection\u0026lt;T\u0026gt; ContainedCollection { get { return _contained; } } /// \u0026lt;summary\u0026gt;Attempts to add the specified value to the end of the deque.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;item\u0026#34;\u0026gt;The item to add.\u0026lt;/param\u0026gt; /// \u0026lt;returns\u0026gt;true if the item could be added; otherwise, false.\u0026lt;/returns\u0026gt; protected virtual bool TryAdd(T item) { return _contained.TryAdd(item); } /// \u0026lt;summary\u0026gt;Attempts to remove and return an item from the collection.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;item\u0026#34;\u0026gt; /// When this method returns, if the operation was successful, item contains the item removed. If /// no item was available to be removed, the value is unspecified. /// \u0026lt;/param\u0026gt; /// \u0026lt;returns\u0026gt; /// true if an element was removed and returned from the collection; otherwise, false. /// \u0026lt;/returns\u0026gt; protected virtual bool TryTake(out T item) { return _contained.TryTake(out item); } /// \u0026lt;summary\u0026gt;Attempts to add the specified value to the end of the deque.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;item\u0026#34;\u0026gt;The item to add.\u0026lt;/param\u0026gt; /// \u0026lt;returns\u0026gt;true if the item could be added; otherwise, false.\u0026lt;/returns\u0026gt; bool IProducerConsumerCollection\u0026lt;T\u0026gt;.TryAdd(T item) { return TryAdd(item); } /// \u0026lt;summary\u0026gt;Attempts to remove and return an item from the collection.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;item\u0026#34;\u0026gt; /// When this method returns, if the operation was successful, item contains the item removed. If /// no item was available to be removed, the value is unspecified. /// \u0026lt;/param\u0026gt; /// \u0026lt;returns\u0026gt; /// true if an element was removed and returned from the collection; otherwise, false. /// \u0026lt;/returns\u0026gt; bool IProducerConsumerCollection\u0026lt;T\u0026gt;.TryTake(out T item) { return TryTake(out item); } /// \u0026lt;summary\u0026gt;Gets the number of elements contained in the collection.\u0026lt;/summary\u0026gt; public int Count { get { return _contained.Count; } } /// \u0026lt;summary\u0026gt;Creates an array containing the contents of the collection.\u0026lt;/summary\u0026gt; /// \u0026lt;returns\u0026gt;The array.\u0026lt;/returns\u0026gt; public T[] ToArray() { return _contained.ToArray(); } /// \u0026lt;summary\u0026gt;Copies the contents of the collection to an array.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;array\u0026#34;\u0026gt;The array to which the data should be copied.\u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;index\u0026#34;\u0026gt;The starting index at which data should be copied.\u0026lt;/param\u0026gt; public void CopyTo(T[] array, int index) { _contained.CopyTo(array, index); } /// \u0026lt;summary\u0026gt;Copies the contents of the collection to an array.\u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;array\u0026#34;\u0026gt;The array to which the data should be copied.\u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;index\u0026#34;\u0026gt;The starting index at which data should be copied.\u0026lt;/param\u0026gt; void ICollection.CopyTo(Array array, int index) { _contained.CopyTo(array, index); } /// \u0026lt;summary\u0026gt;Gets an enumerator for the collection.\u0026lt;/summary\u0026gt; /// \u0026lt;returns\u0026gt;An enumerator.\u0026lt;/returns\u0026gt; public IEnumerator\u0026lt;T\u0026gt; GetEnumerator() { return _contained.GetEnumerator(); } /// \u0026lt;summary\u0026gt;Gets an enumerator for the collection.\u0026lt;/summary\u0026gt; /// \u0026lt;returns\u0026gt;An enumerator.\u0026lt;/returns\u0026gt; IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); } /// \u0026lt;summary\u0026gt;Gets whether the collection is synchronized.\u0026lt;/summary\u0026gt; bool ICollection.IsSynchronized { get { return _contained.IsSynchronized; } } /// \u0026lt;summary\u0026gt;Gets the synchronization root object for the collection.\u0026lt;/summary\u0026gt; object ICollection.SyncRoot { get { return _contained.SyncRoot; } } Wow! Thats a fair bit of code in C#, fair enough there is a lot of noise in the xml doc comments, but theres also a lot of boiler plate code in there too.\nOk now we have gotten that out of the way heres the good bit. Below is an agent based design which implements the same functionality but uses a lot less code.\nmodule Poc //Agent alias for MailboxProcessor type Agent\u0026lt;\u0026#39;T\u0026gt; = MailboxProcessor\u0026lt;\u0026#39;T\u0026gt; ///One of three messages for our Object Pool agent type PoolMessage\u0026lt;\u0026#39;a\u0026gt; = | Get of AsyncReplyChannel\u0026lt;\u0026#39;a\u0026gt; | Put of \u0026#39;a * AsyncReplyChannel\u0026lt;unit\u0026gt; | Clear of AsyncReplyChannel\u0026lt;List\u0026lt;\u0026#39;a\u0026gt;\u0026gt; /// Object pool representing a reusable pool of objects type ObjectPool\u0026lt;\u0026#39;a\u0026gt;(generate: unit -\u0026gt; \u0026#39;a, initialPoolCount) = let initial = List.init initialPoolCount (fun (x) -\u0026gt; generate()) let agent = Agent.Start(fun inbox -\u0026gt; let rec loop(x) = async { let! msg = inbox.Receive() match msg with | Get(reply) -\u0026gt; let res = match x with | a :: b -\u0026gt; reply.Reply(a);b | [] as empty-\u0026gt; reply.Reply(generate());empty return! loop(res) | Put(value, reply)-\u0026gt; reply.Reply() return! loop(value :: x) | Clear(reply) -\u0026gt; reply.Reply(x) return! loop(List.empty\u0026lt;\u0026#39;a\u0026gt; ) } loop(initial)) /// Clears the object pool, returning all of the data that was in the pool. member this.ToListAndClear() = agent.PostAndAsyncReply(Clear) /// Puts an item into the pool member this.Put(item) = agent.PostAndAsyncReply((fun ch -\u0026gt; Put(item, ch))) /// Gets an item from the pool or if there are none present use the generator member this.Get(item) = agent.PostAndAsyncReply(Get) We have a discriminated union (PoolMessage) which describes the messages that we are going to use with this agent, they are pretty straight forward to follow. Get simply returns either a stored item or generates a brand new one using the generator function which is passed into the ObjectPools constructor (generate: unit -\u0026gt; \u0026lsquo;a). Put simply adds the item onto the internal list. Clear simply returns the current pool and then clears it. The core processing all happens in the async{} block, we simply wait for a message to arrive, then we pattern match on one of the messages either Get,Put, or Clear.\nGet takes an item from the internal list if there are items present, otherwise it invokes the generator function and returns a newly generated object.\nFor a Put operation we use the cons (::) operator to add the item onto the internal list via the recursive loop.\nFor the Clear operation we return the entire list then return an empty list to the recursive loop.\nI think you will agree this is a nice succinct example of the flexibility and elegance of agents and yet another reason to use F# for more server side activities. It\u0026rsquo;s not simply a language for the mathematical and finance orientated developers.\nFor anyone interested all of the code should be in my GitHub repository to download.\nThanks to Tomas Petricek for suggesting using the recursive loop to pass the list rather than using a ref cell and the (:=) operator.\nUntil next time\u0026hellip;\n","date":"June 5, 2011","externalUrl":null,"permalink":"/programming/2011-06-05-agents-and-objectpools/","section":"Blog","summary":"Everyone knows F# agents are cool right? Well here’s yet another example of how versatile they can be…\nThere was a series of posts last April by Stephen Toub from the pfxteam at Microsoft. I was reading through some of the posts again the other day and thought some of the ideas presented there would make interesting projects in F# to demonstrate the flexibility and succinctness of the language. I thought the ObjectPool example would make an interesting project in F# using agents aka MailboxProcessors. An ObjectPool is basically a pool of objects that have been pre-created so that you can grab one and use it, and then place it back in the pool when you’re finished. They are useful in situations where the cost of creating object from scratch is very high or you want to cut down on allocations in the garbage collector.\n","title":" Agents and ObjectPools","type":"programming"},{"content":"Ok so I have been offline for a while now, what with starting a new financial contract in London and not having any broadband access for a while. I have been working on something, honest!\nSince the last post I have been reflecting on the pipeline design and it had a distinct object orientated feel to it that I wasnt happy with, so I have amended the structure of the code and come up with the following which simplifies in some areas and expands in others\u0026hellip;\nmodule Pipeline open System.Collections.Concurrent [\u0026lt;Interface\u0026gt;] type IPipelineInput\u0026lt;\u0026#39;a\u0026gt; = abstract Insert: \u0026#39;a -\u0026gt; unit [\u0026lt;Interface\u0026gt;] type IPipelineConnection\u0026lt;\u0026#39;a\u0026gt; = abstract Attach: IPipelineInput\u0026lt;\u0026#39;a\u0026gt; -\u0026gt; unit abstract Detach: IPipelineInput\u0026lt;\u0026#39;a\u0026gt; -\u0026gt; unit [\u0026lt;Interface\u0026gt;] type IPipeline\u0026lt;\u0026#39;a,\u0026#39;b\u0026gt; = inherit IPipelineConnection\u0026lt;\u0026#39;b\u0026gt; inherit IPipelineInput\u0026lt;\u0026#39;a\u0026gt; type PipelineStage\u0026lt;\u0026#39;a,\u0026#39;b\u0026gt;(processor, router: seq\u0026lt;IPipelineInput\u0026lt;\u0026#39;b\u0026gt;\u0026gt; * \u0026#39;b -\u0026gt; seq\u0026lt;IPipelineInput\u0026lt;\u0026#39;b\u0026gt;\u0026gt;, ?overflow, ?capacity, ?blockingTime) = let processor = processor let router = router let createBlockingCollection x = match x with | Some c -\u0026gt; new BlockingCollection\u0026lt;\u0026#39;a\u0026gt;(c:int) | None -\u0026gt; new BlockingCollection\u0026lt;\u0026#39;a\u0026gt;() let buffer = createBlockingCollection capacity let routes = ref List.empty\u0026lt;IPipelineInput\u0026lt;\u0026#39;b\u0026gt;\u0026gt; let queuedOrRunning = ref false let blocktime = match blockingTime with | Some b -\u0026gt; b | None -\u0026gt; 250 let consumerLoop = async { try let rec loop()= let item = ref Unchecked.defaultof\u0026lt;_\u0026gt; let taken = buffer.TryTake(item, blocktime) if taken then do !item |\u0026gt; processor |\u0026gt; Seq.iter (fun z -\u0026gt; (match !routes with | [] -\u0026gt; ()(*we cant route with no routes*) | _ -\u0026gt; do router (!routes, z) |\u0026gt; Seq.iter (fun r -\u0026gt; (r.Insert z ))) ) loop() else ()(*exit nothing to consume in time limit*) loop() with e -\u0026gt; raise e } member this.ClearRoutes = routes := [] interface IPipelineInput\u0026lt;\u0026#39;a\u0026gt; with member this.Insert payload = let added = buffer.TryAdd(payload, blocktime) if added then //begin consumer loop if not !queuedOrRunning then lock consumerLoop (fun() -\u0026gt; Async.Start(async {do! consumerLoop }) queuedOrRunning := true) else() else //overflow here if function passed match overflow with | Some t -\u0026gt; payload |\u0026gt; overflow.Value | None -\u0026gt; () interface IPipelineConnection\u0026lt;\u0026#39;b\u0026gt; with member this.Attach (stage) = let current = !routes routes := stage :: current member this.Detach (stage) = let current = !routes routes := List.filter (fun el -\u0026gt; el \u0026lt;\u0026gt; stage) current static member Attach (a:IPipelineConnection\u0026lt;_\u0026gt;) (b) = a.Attach b ;b static member Detach (a: IPipelineConnection\u0026lt;_\u0026gt;) (b) = a.Detach b ;a static member (++\u0026gt;) (a:IPipelineConnection\u0026lt;_\u0026gt;, b) = a.Attach (b) ;b static member (--\u0026gt;) (a:IPipelineConnection\u0026lt;_\u0026gt;, b) = a.Detach b ;a static member (\u0026lt;\u0026lt;--) (a:IPipelineInput\u0026lt;_\u0026gt;, b:\u0026#39;b) = a.Insert b static member (--\u0026gt;\u0026gt;) (b,a:IPipelineInput\u0026lt;_\u0026gt;) = a.Insert b Summary. # I only want to summarise the code as I think its fairly straight forward to see whats going on.\nInterfaces # We have two main interfaces defined IPipelineInput\u0026lt;\u0026lsquo;a\u0026gt; and **IPipelineConnection\u0026lt;\u0026lsquo;a\u0026gt;, **as you can tell by the names they are involved with connecting the pipeline together and getting information into the pipeline. Those two interfaces are merged together in the IPipeline\u0026lt;\u0026lsquo;a, \u0026lsquo;b\u0026gt; interface, this keeps a nice separation between connecting and inserting into the pipeline, it also makes implementation easier and allows the interfaces to be implemented in other areas of code that need to talk to or connect to a pipeline.\nInternals # Inside the pipeline we have the bounded blocking queue which is implemented by the BlockingCollection from TPL. This is used to store the pipeline payloads that are waiting to be processed.\nThe consumerLoop function is recursive and continually tries to take items from the blocking collection processing and routing each one to the next pipeline stage.\nThe processor is a function that transforms from type \u0026lsquo;a to type \u0026lsquo;b.\nThe router is a function that takes a sequence of IPipelineInput\u0026lt;\u0026lsquo;b\u0026gt; and also the payload \u0026lsquo;b it returns a sequence of IPipelineInput\u0026lt;\u0026lsquo;b\u0026gt;. What this effectively means is that we can route by the connected stages (i.e. round robin routing, multi-cast routing.) Or we could route by payload contents (i.e. if the payload contains a certain bytes sequence we could choose a certain IPipelineInput\u0026lt;\u0026lsquo;b\u0026gt;.)\nEach item taken is passed to the processor and router via pipeline (|\u0026gt;) and Seq operations, recursively calling itself until an item can no longer be retrieved from the buffer.\nThe implementation of IPipelineInput\u0026lt;\u0026lsquo;a\u0026gt;.Insert is the counterpart to the previous function. It first tries to inset the item into the bounded blocking queue, if this cannot be done then the overflow function is called if one is present. Next the async consumer loop is started if it is not already running. The idea behind this is that by keeping the payload processing running on the thread pool while there is work to do it will cut down on the number of context switches between threads. Once an item cannot be taken from the bounding blocking queue the loop will exit.\nThe rest of the code is pretty standard stuff and should be pretty easy to follow.\nI also define some symbolic operations to simply constructing and using the pipeline:\n++\u0026gt; Attaches the pipeline stage on the right hand side to the one on the left. \u0026ndash;\u0026gt; Detaches the pipelinestage on the right from the one on the left. \u0026laquo;\u0026ndash; Inserts a payload on the right into the pipeline stage on the left. \u0026ndash;\u0026raquo; Inserts a payload on the left hand side into the pipeline stage on the right.\nThese help to keep a nice terse description of the pipeline, once things get a little more complex other operators may be required, the now discontinued Axiom had a whole host of these, its a pity Microsoft dropped the language.\nExample # Heres a quick sample pipeline showing the pipeline in use:\nStage 1 takes a string and splits it based on the \u0026lsquo;,\u0026rsquo;. Stage 2 reverses each string. Stage 3 reverses the string back to the original. module program open System open Pipeline let consoleLock = new obj() let split del n (s:string) = lock consoleLock (fun() -\u0026gt; do printfn \u0026#34;%A:before split %A\u0026#34; n s let split = s.Split([|del|]) do printfn \u0026#34;%A:after: split into: %A\u0026#34; n split split |\u0026gt; Array.toSeq) let reverse (s:string) = new string(s |\u0026gt; Seq.toArray |\u0026gt; Array.rev) let oneToSingleton a b f= lock consoleLock (fun() -\u0026gt; printfn \u0026#34;%A:before reverse %A\u0026#34; a b let result = b |\u0026gt; f printfn \u0026#34;%A:after reverse %A\u0026#34; a result result|\u0026gt; Seq.singleton) let OneToSeqRev a b = oneToSingleton a b reverse ///Simply picks the first route let basicRouter( r, i) = let head = Seq.head r Seq.singleton head let p1 = PipelineStage( split \u0026#39;,\u0026#39; \u0026#34;1\u0026#34;, basicRouter) let p2 = PipelineStage( OneToSeqRev \u0026#34;2\u0026#34;, basicRouter) let p3 = PipelineStage( OneToSeqRev \u0026#34;3\u0026#34;, basicRouter) p1 ++\u0026gt; p2 ++\u0026gt; p3 |\u0026gt; ignore let generateCircularSeq (lst:\u0026#39;a list) = let rec next () = seq { for element in lst do yield element yield! next() } next() for str in [\u0026#34;John,Paul,George,Ringo\u0026#34;] |\u0026gt; generateCircularSeq |\u0026gt; Seq.take 10 do str --\u0026gt;\u0026gt; p1 let x = Console.ReadKey() As you can see the assignment of the pipeline stages is pretty simple as is the composition of multiple stages. This was often one of the most difficult areas while developing a similar pipelines in C# you could often find yourself with a few hundred lines of setup code which was a often a nightmare to debug a few weeks later.\nHopefully I have whet your appetite with pipelines, in a future article I will be combining socket operations with pipeline stages to produce a flexible framework to deal with high throughput network applications.\nAs always I appreciate any comments, until next time\u0026hellip;\n","date":"April 4, 2011","externalUrl":null,"permalink":"/programming/2011-04-04-pipeline-processing-3/","section":"Blog","summary":"Ok so I have been offline for a while now, what with starting a new financial contract in London and not having any broadband access for a while. I have been working on something, honest!\nSince the last post I have been reflecting on the pipeline design and it had a distinct object orientated feel to it that I wasnt happy with, so I have amended the structure of the code and come up with the following which simplifies in some areas and expands in others…\n","title":"Pipeline processing 3","type":"programming"},{"content":"","date":"April 4, 2011","externalUrl":null,"permalink":"/series/pipelineprocessing/","section":"Series","summary":"","title":"Pipelineprocessing","type":"series"},{"content":"","date":"April 4, 2011","externalUrl":null,"permalink":"/tags/sockets/","section":"Tags","summary":"","title":"Sockets","type":"tags"},{"content":" Welcome to pipeline processing part 2. # I feel I need to backtrack slightly from the previous post, having worked with pipelines for quite some time I have the advantage of knowing all of the details that may be alluded to in these articles without being effected by any omissions I may make, obviously you guys aren\u0026rsquo;t in that position, so I\u0026rsquo;m going to try and rectify that a bit now. If you have any queries then please leave a comment and I will try to address them in further articles. Pipelines are a simple concept but in practice there can be some caveats and things to bear in mind, sometime the whole mindset of development team can be against them unless they can see the bigger picture\u0026hellip;\nFirst of all one of the most important things to bear in mind with a pipeline is that you are only going to be as fast as your slowest stage, if one stage is ten times slower than another then it will be waiting for input most of the time, we need to make this more efficient.\nPremature Optimisation # Lots of developers out there have the premature optimisation is the root of all evil mindset and will quote this out loud to you when you mention performance early on. I\u0026rsquo;m not advocating premature optimisation, in this instance performance is key, if one stage is out of kilter with the rest then we are going to be running at that pace of the slowest stage, if that\u0026rsquo;s too slow for the requirements then you are screwed.\nThe more I think about performance the more I believe its an essential part of creating code. There are too many developers these days that will produce sloppy unrefined plain bad code. I\u0026rsquo;m a keen believer in producing quality code that you can be proud of, and part of that is having clean code that\u0026rsquo;s both efficient and works. I think some of this boils down to a feature driven approach that measures developers solely in terms of features added, take the typical burn down chart that you would use in agile software development:\nThere is nowhere on this chart that measures whether the code is good or bad or runs to performance requirements. In the future I may do an article on integrating code quality into your build process, its something I have been thinking about doing for a while now.\nWhile I\u0026rsquo;m talking about performance you also might want to check out Joe Duffy\u0026rsquo;s post on [The \u0026lsquo;premature optimization is evil\u0026rsquo; myth](http://www.bluebyt esoftware.com/blog/2010/09/06/ThePrematureOptimizationIsEvilMyth.aspx), and also check out Joe\u0026rsquo;s book on [concurrent programming](http://www.bluebytesoftw are.com/books/winconc/winconc_book_resources.html), put it on your wish list if you haven\u0026rsquo;t already read it, its a great book.\nUnbalanced pipelines # Data is received from the network via packets, each packet may contain one or more messages from a business systems or indeed a partial message. We need to collect the packets either separate or combine them to form individual messages, deserialize them and finally log them.\nHere\u0026rsquo;s a sample pipeline demonstrating an unbalanced pipeline:\nStage 1 of the pipeline receives these packets and processes them into individual messages passing them onto Stage 2. We now have a complete message (in this instance the message will be XML) we want to turn it into a .Net type we now deserialize the message and pass it onto Stage 3. To keep this pipeline simple all we are going to do here is log type of message to disk or a database, the pipeline is now complete. Stage 1 would take 5 seconds to fully utilise stage 2, stage 2 would take 2 seconds to fully utilise stage 3. You can see this pipeline will only process 100 transactions per second even though stages 2 has 5x the throughput of stage 1 and stage 3 has 2x the throughput of stage 2. Our efficiency is only about 10% of what it could be, we must be able to do something about that.\nLets look at the following diagram which demonstrate a balanced pipeline:\nBalanced pipelines # You can see from this diagram that each stage processes the same number of transactions per second by introducing parallel stages. This is called a balanced pipeline. Sometimes you cant get a perfectly balanced pipeline but you should strive to get as close as possible. Sometimes a certain stage cannot be parallelised because it may have mutable state, or you are using some sort of IOC container for processing services, this might make constructing the various stages in parallel difficult, this can become an art form in itself and can lead to very large initialisation sections in the code. I hope to address all of these issues in due course.\nThis poses some interesting thoughts and questions to add to some you may already have:\nHow can we easily manage the complexity of parallelism? How will the distribution of work be handled? How do you baseline the throughput of each stage? Can you automate the parallelism of a particular stage? How do you manage the complexity of multiple stages? What about parallelism and mutable state? The final point to note is the Distributor/Router must operate at a much higher rate than the processing stages otherwise you will introduce another bottle neck into the system, although you could have a multiple distributors but this would yet another degree of complexity that has to be managed. You can see that things can quickly become more complicated than they first seemed.\nI know I promised lots of funky code but I figured there was a bit more explaining to do before we can get to that. I want to take a more of an iterative approach to show you the potential pitfalls that can occur during developing such a pipeline and how to avoid them. I thought this would be a lot more constructive than dropping a load of code and some pretty pictures and hoping for the best.\nNext time we will be exploring a simple pipeline stage with a single degree of parallelism and a simple router. After that we will then start exploring and answering the questions above, adding more features like parallelism, instrumentation, and visualisation.\nHope you enjoyed this even though there was no code!\nSee you next time.\n","date":"February 13, 2011","externalUrl":null,"permalink":"/programming/2011-02-13-pipeline-processing-2/","section":"Blog","summary":"Welcome to pipeline processing part 2. # I feel I need to backtrack slightly from the previous post, having worked with pipelines for quite some time I have the advantage of knowing all of the details that may be alluded to in these articles without being effected by any omissions I may make, obviously you guys aren’t in that position, so I’m going to try and rectify that a bit now. If you have any queries then please leave a comment and I will try to address them in further articles. Pipelines are a simple concept but in practice there can be some caveats and things to bear in mind, sometime the whole mindset of development team can be against them unless they can see the bigger picture…\n","title":"Pipeline processing 2","type":"programming"},{"content":" Welcome to new series of articles on pipeline processing. # First up, what\u0026rsquo;s a pipeline? Well according to Wikipedia:\nA pipeline is a set of data processing elements connected in series, so that the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion; in that case, some amount of buffer storage is often inserted between elements.\nIn essence its a way of dealing with complexity and its also a way of breaking down a process into separate tasks of a similar size. If they are used correctly then pipelines can be used to increase the overall throughput of a system.\nIn enterprise systems or in fact in most large systems, a simple idea or program can rapidly become overwhelmingly complex. The management all of the disparate parts of the system can become a nightmare and the code can quickly becomes a labyrinth, navigating it becomes a skill of only the most accomplished code _ninja, _and even then your playing Russian roulette with any bug fixes.In an effort to keep things manageable and simple one approach that we can use is a pipeline. The idea is that each stage is connected to one or more other stages and each that each stage deals with a single task before passing the work onto the next stage. There are many primitive types in the Task Parallel Library (TPL) that you could use to compose a working pipeline, we will be using a lightweight subset taking only a few core ideas and making sure we get a nice slick design that is both powerful and flexible.\nHere\u0026rsquo;s a quick flow diagram of the sort of thing that we will be looking at:\nThis is a generic asynchronous payload based pipeline. Each stage is asynchronous and self contained and is connected to one or more other stages. As a payload enters the pipeline it is initially added to a bounding blocking queue. If the queue is full then the payload is said to have overflowed and is passed to the failure processor where the payload can be processed or transformed in some way before being passed to a failure router which would in turn pass the payload to one or more of the next failure stages. The same is also true for a successfully queued payload except that the payload is first dequeued, processed, then passed to a router which then passes the payload to one or more stages. If an exception occurs during processing then the payload is passed to the failure processor and processed like an overflow. I am purposely missing out any details of asynchronous operation as they will be described in more detail next time.\nWe will be using a little bit of Language Oriented Programming to construct the pipeline stages, maybe using a little bit of operator overloading too. I will describe all of this in more detail next time as we dig into the code. I want this to be just a brief introduction to what we are going to be doing.\nHere\u0026rsquo;s a more detailed description of the components that are involved in each stage:\nBounded Blocking Queue # This is a standard bounded blocking queue from the TPL, its purpose here is to limit the amount of payloads that are waiting to be processed, each queue will have an associated time-out period, if the time-out period passes the payload is passed to the failure processor for processing and then finally to the the failure router to be passed to one or more failure stages.\nProcessors # Each pipeline processor has a primary Processor\u0026lt;T,U\u0026gt; and a failure processor\u0026lt;T,V\u0026gt;.\nThe primary processors job is to convert type T to type U, both types can be the same if you wish, you may well be thinking why would I want a processing stage that essentially leaves the type unchanged? In this case the processor acts as a simple a pass through but using this you to do some custom routing. This can be very be useful in some scenarios and I will describing this in more detail in a further post.\nEach pipeline stage also has a failure processor\u0026lt;T,V\u0026gt;. The failure processor acts on the payload to produce the desired type and passes it onto the failure router. The reasoning behind this scheme rather than a simplistic exception logger is simply flexibility. Having spent a lot of time with this kind of API in a more locked down format I have found that you can end up wanting a bit more flexibility especially when some developers try to get a bit creative with the API or start state to the payload. A good example of having some flexibility is during overflow: If the bounded blocking queue fills up and blocks for the time-out period then the payload could be passed to a failure failure processor in which types T and V are the same. This would allow us to pass the payload to another stage and retry later on by attaching some sort of delayed forwarding pipeline stage.\nRouters # The router is responsible for getting the payload to the next pipeline stage, it can be implemented as a simple predicate function operating on the type directly or even some outside influence if you wish. An example of this might be a simple duplicating stage where the payload is passed to multiple output stages rather than just one, or a time based router where one stage is passed the payload during the day and another at night. When you start to think about the possibilities the Processor / Router combination can be really really flexible.\nEach pipeline stage also has a corresponding has a failure router, this can be used for all sorts of purposes like routing the failed payload to a logging component, routing to a delayed retry mechanism, or saved to a database etc.\nThats all for now, we will be digging into some code and more detail next time, and I will be describing a few different types of pipelines so you can get a feel of how to use them and the overall structure.\nAnother interesting aspect of these pipelines is that once constructed they can be composed into single reusable blocks that as a whole, represent a single pipeline stage. These composite stages can then be connected together to form a super pipeline stage, complexity is only visible when you start to drill down and becomes almost fractal like\u0026hellip;\nAs always please leave any comments or suggestions.\n","date":"February 1, 2011","externalUrl":null,"permalink":"/programming/2011-02-01-pipeline-processing-1/","section":"Blog","summary":"Welcome to new series of articles on pipeline processing. # First up, what’s a pipeline? Well according to Wikipedia:\nA pipeline is a set of data processing elements connected in series, so that the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion; in that case, some amount of buffer storage is often inserted between elements.\n","title":"Pipeline processing 1","type":"programming"},{"content":" Welcome to part 4 # If you were looking forward to some exciting new F# code this time your going to be disappointed, however if you are like me and like looking at graphs and stats and digging in deeper into the code then your going to enjoy this, lets get started\u0026hellip;\nI set up a 5 minute test with 50 clients connecting to the server with a 15ms interval between each one. Once connected each client receives a 128 byte message from the server every 100ms so this will be a 500 messages per second test. I am going to be using an excellent product called YourKit Profilerfor .NET it can do both memory and CPU profiling as well as displaying telemetry for things like thread count, stack contents, memory allocations etc. It can be configured to be a lot less intrusive than a lot of other profilers and I have had a lot of success using it. You can download a demo from their site using the link above. I will be doing some other articles on using profiling and analysis tools later on so stay tuned for those too. All of the graphs and information gathered in this post come from YourKits output during CPU and memory profiling.\nBefore we start here\u0026rsquo;s a reminder of what the client code looks like, this is a simple test client using Brian\u0026rsquo;s code as mentioned in Part1 I have highlighted the lines that have changed below.\nopen System.Net open System.Net.Sockets let quoteSize = 128 type System.Net.Sockets.TcpClient with member client.AsyncConnect(server, port, clientIndex) = Async.FromBeginEnd(server, port,(client.BeginConnect : IPAddress * int * _ * _ -\u0026gt; _), client.EndConnect) let clientRequestQuoteStream (clientIndex, server, port:int) = async { let client = new System.Net.Sockets.TcpClient() do! client.AsyncConnect(server,port, clientIndex) let stream = client.GetStream() let! header = stream.AsyncRead 1 // read header while true do let! bytes = stream.AsyncRead quoteSize if Array.length bytes \u0026lt;\u0026gt; quoteSize then printfn \u0026#34;client incorrect checksum\u0026#34; } let myLock = new obj() let clientAsync clientIndex = async { do! Async.Sleep(clientIndex*15) if clientIndex % 10 = 0 then lock myLock (fun() -\u0026gt; printfn \u0026#34;%d clients...\u0026#34; clientIndex) try do! clientRequestQuoteStream (clientIndex, IPAddress.Loopback, 10003) with e -\u0026gt; printfn \u0026#34;CLIENT %d ERROR: %A\u0026#34; clientIndex e //raise e } Async.Parallel [ for i in 1 .. 50 -\u0026gt; clientAsync i ] |\u0026gt; Async.Ignore |\u0026gt; Async.Start System.Console.ReadKey() |\u0026gt; ignore CPU and threading performance # First of all lets look at the CPU results from the IAsync pattern:\nHeres the same run from the SAEA pattern:\nYou can see that both the number of threads and the amount of CPU is a quite a lot less in the SAEA pattern. The spike at the beginning is the allocation of buffers for the BocketPool.\nNow lets move on to memory and garbage collection.\nMemory Allocation # Here\u0026rsquo;s a graph of the heap and process memory allocation in the IAsync pattern, green is generation 0, blue is generation 1 and orange is the large object heap. There\u0026rsquo;s also red for generation 2 but the results are behind the others and they are only small 0,2 MB peaks at 5 to 15 second intervals.\nHeres the same but for the SAEA pattern, there are red peaks every 10- 20 second intervals of 0.2MB hidden under the others.\nAs you can see the heap memory is around half the size and the process memory is 15MB less.\nMemory Hotspots # Finally here\u0026rsquo;s a couple of screen shot of the hot spots for memory allocations in both implementations\nIAsync SAEA You can clearly the IAsync allocations are not present in the SAEA implementation and there are 310,188 of them, that\u0026rsquo;s 27% of the total garbage!\nFinal thoughts # The SAEA pattern definitely cuts down on memory and CPU usage, yes it adds a lot of complexity but if your application is dealing with a very high volume of traffic or clients and you need optimal performance then I think its the way to go.\nThe optimisations don\u0026rsquo;t stop there either, if you think about it the receive Bocketpool is not even used here, if we collapsed all of the BocketPools into a single contiguous store then we would use even less resources, this means we could support even more clients or throughput. In a typical high volume scenario you are looking at doubling your throughput or number of client connections.\nThere\u0026rsquo;s definitely a lot more or interesting things to explore in this area.\nAs usual any comments are welcome.\nSee you next time\u0026hellip;\n","date":"January 28, 2011","externalUrl":null,"permalink":"/programming/2011-01-28-sockets-and-bockets-part-4/","section":"Blog","summary":"Welcome to part 4 # If you were looking forward to some exciting new F# code this time your going to be disappointed, however if you are like me and like looking at graphs and stats and digging in deeper into the code then your going to enjoy this, lets get started…\n","title":"Sockets and Bockets 4","type":"programming"},{"content":"","date":"January 28, 2011","externalUrl":null,"permalink":"/series/socketsandbockets/","section":"Series","summary":"","title":"SocketsAndBockets","type":"series"},{"content":" Welcome to part three! # As promised heres a description of the inner workings. I\u0026rsquo;m sick to death of typing SocketAsyncEventArgs so from now on I will refer to it as SAEA.\nBocketPool\nThe BocketPool has an interesting name and with it an interesting constructor! It takes the following parameters:\nnumber: The number of items to create in the BocketPool. size: The size of each buffer in bytes. callback: A callback function which is invoked whenever the SAEA object completes its operation.\ntype BocketPool( number, size, callback) as this = let number = number let size = size let totalsize = (number * size) let buffer = Array.create totalsize 0uy let pool = new BlockingCollection\u0026lt;SocketAsyncEventArgs\u0026gt;(number:int) A buffer is created with a size equal to the (number * size) in bytes.\ndo let rec loop n = match n with | x when x \u0026lt; totalsize -\u0026gt; let saea = new SocketAsyncEventArgs() saea.Completed |\u0026gt; Observable.add callback saea.SetBuffer(buffer, n, size) this.CheckIn(saea) loop (n + size) | _ -\u0026gt; () loop 0 The tail recursive loop function creates a SAEA object and adds it to the BlockingCollection(pool).\nThe buffer is assigned to each SAEA but each is given a unique offset to use, this is done by the SetBuffer method. Using this method of allocation, memory fragmentation is reduced to a minimum by allowing the same buffer to be reused.\nWe use the pipeline operator to attach the Completed event to the callback method that is passed in the constructor.\nThe CheckIn, CheckOut, and Count methods are simply wrappers around the BlockingCollection.\nWe also implement IDisposable to take care of the disposal of the SAEA in the BlockingCollection.\nConnection\nThe main purpose for this type is to encapsulate the sending and receiving of messages for a particular client. A BocketPool is created for both the send and receive operations, the receiveCompleted and SentCompleted are invoked when the respective operations complete.\nSend\nmember this.Send (msg:byte[]) = let s = sendPool.CheckOut() Buffer.BlockCopy(msg, 0, s.Buffer, s.Offset, msg.Length) socket.SendAsyncSafe(this.sendCompleted, s) Initially a Bocket is checked out of the sendPool using sendPool.Checkout(), the msg byte array is copied to the corresponding Offset property of the SAEA.\nFinally the SendAsyncSafe extension method is called passing in the SAEA and the callback.\nsendCompleted\nmember this.sendCompleted (args: SocketAsyncEventArgs) = try match args.LastOperation with | SocketAsyncOperation.Send -\u0026gt; match args.SocketError with | SocketError.Success -\u0026gt; () | SocketError.NoBufferSpaceAvailable | SocketError.IOPending | SocketError.WouldBlock -\u0026gt; if not(anyErrors) then anyErrors \u0026lt;- true failwith \u0026#34;Buffer overflow or send buffer timeout\u0026#34; | _ -\u0026gt; args.SocketError.ToString() |\u0026gt; printfn \u0026#34;socket error on send: %s\u0026#34; | _ -\u0026gt; failwith \u0026#34;invalid operation, should be receive\u0026#34; finally sendPool.CheckIn(args) This function matches the LastOperation property of the SAEA using pattern matching, this ensures that the LastOperation is always SocketError.Success.\nWe raise exceptions on NoBufferSpaceAvailable, IOPending, and WouldBlock as buffer overflows and match any other conditions the wildcard.\nFinally we Check the Bocket back in so that it can be reused.\nreceiveCompleted\nmember this.receiveCompleted (args: SocketAsyncEventArgs) = try match args.LastOperation with | SocketAsyncOperation.Receive -\u0026gt; match args.SocketError with | SocketError.Success -\u0026gt; socket.ReceiveAsyncSafe( this.receiveCompleted, receivePool.CheckOut()) let data = Array.create args.BytesTransferred 0uy Buffer.BlockCopy(args.Buffer, args.Offset, data, 0, data.Length) let client = args.RemoteEndPoint args.RemoteEndPoint \u0026lt;- null data |\u0026gt; printfn \u0026#34;received data: %A\u0026#34; | _ -\u0026gt; args.SocketError.ToString() |\u0026gt; printfn \u0026#34;socket error on receive: %s\u0026#34; | _ -\u0026gt; failwith \u0026#34;unknown operation, should be receive\u0026#34; finally receivePool.CheckIn(args) This function is very similar to the sendCompleted and could probably be refactored a bit using the [Hole in the middle pattern](http://enfranchisedmind.com/blog/posts/the-hole-in-the-middle- pattern/). Again we check to ensure the last operation was a success, we checkout another Bocket and start another ReceiveAsyncSafe. This ensures that the socket can begin another receive operation as soon as possible while we take the data from the SAEA Buffer, we do this with Buffer.Block copy.\nIf this were a fully-fledged API then we would raise an event here so that users of the component could consume the data.\nIn my own component the data is inserted into a series of processing stages using the [Pipeline Pattern](http://www.cise.ufl.edu/research/ParallelPatterns /PatternLanguage/AlgorithmStructure/Pipeline.htm), which I will be may describe in a future post if anyone\u0026rsquo;s interested.\nTcpListener\nThe TcpListener is very similar to the Connection object in that it has a pool of SAEA objects that are used to accept connection from clients, again a round of refactoring could be done here to avoid duplication with the Connection type. The main difference is that we don\u0026rsquo;t need to use the Buffer on the SAEA to send anything to the client when it initially connects.\nacceptCompleted\nmember this.acceptcompleted (args : SocketAsyncEventArgs) = try match args.LastOperation with | SocketAsyncOperation.Accept -\u0026gt; match args.SocketError with | SocketError.Success -\u0026gt; listeningSocket.AcceptAsyncSafe( this.acceptcompleted, acceptPool.Take()) //create new connection let connection = newConnection args.AcceptSocket connection.Start() //update stats reportConnections //async start of messages to client startSending connection //remove the AcceptSocket because we will be reusing args args.AcceptSocket \u0026lt;- null | _ -\u0026gt; args.SocketError.ToString() |\u0026gt; printfn \u0026#34;socket error on accept: %s\u0026#34; | _ -\u0026gt; args.LastOperation |\u0026gt; failwith \u0026#34;Unknown operation, should be accept but was %a\u0026#34; finally acceptPool.Add(args) This function is similar to the send and receive completed methods in the Connection type, although this time we create a Connection object and call the Start function, this puts the Connection into receive mode.\nThe reportConnections is called next which simply prints how many clients are connected, we now start an Asyncronous workflow using the startSending function.\nFinally we set the AcceptSocket property to null on the SAEA object and add it back to the BlockingCollection so that it can be reused.\nThe purpose of the BlockingCollection here is to have a fixed pool of SAEA that block when there isn\u0026rsquo;t an SAEA to service the new connection, this could be a potential issue for the client as it could timeout while waiting for a connection but this is a far preferable situation than causing your server to be effectively denied service due to overload.\nstartSending\nlet startSending connection = Async.Start (async { try use _holder = connection do! asyncServiceClient connection with e -\u0026gt; if not(anyErrors) then anyErrors \u0026lt;- true Console.WriteLine(\u0026#34;server ERROR\u0026#34;) raise e } ) This function uses the syntactic sugar of the asynchronous workflows to start an operation on the Thread pool, once queued on the thread pool it is wrapped in a using block with the _use holder = connection statement and asynchronously calls the asyncServiceClient function, this has the effect of disposing of the Connection type when it exits scope or encounters an exception.\nasyncServiceClient\nlet asyncServiceClient (client: Connection) = async { client.Send(header) while true do do! asyncWriteStockQuote(client) } This function sends a one byte header message to the client using the Connection.Send, followed by calling asyncWriteStockQuote in a continuous loop.\nasyncWriteStockQuote\nlet asyncWriteStockQuote(connection:Connection) = async { do! Async.Sleep 1000 connection.Send(testMessage) Interlocked.Increment(\u0026amp;numWritten) |\u0026gt; ignore } This function sleeps for 1000ms and uses the Connection.Send function to sent a message to the client, the number of results is updated using the Interlocked class.\nI would like to refer you to Brian McNamara\u0026rsquo;s post that describes this part in more detail. The only difference in our workflow is that we don\u0026rsquo;t use a stream operation as we have the SendAsyncSafe function to do all the work for us. IDispose is also implemented on this type too as we have to dispose of the SAEA objects that are used for the asynchronous accepts.\ncreateTcpSocket\nlet createTcpSocket() = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp) This function simply wraps the Sockets class constructor mapping it to: Tcp protocol, Streaming, and InterNetwork Address type.\ncreateListener\nlet createListener (ip:IPAddress, port, backlog) = let s = createTcpSocket() s.Bind(new IPEndPoint(ip, port)) s.Listen(backlog); s This function calls the createTcpSocket function, binds to the IPAddress and port that are passed in and starts listening for connections.\nStart\nmember this.start () = listeningSocket.AcceptAsyncSafe( this.acceptcompleted, acceptPool.Take()) while true do Thread.Sleep 1000 let count = Interlocked.Exchange(\u0026amp;numWritten, 0) count |\u0026gt; printfn \u0026#34;Quotes per sec: %A\u0026#34; This function starts the whole process of listening for a connection from clients. A SAEA is taken from the BlockingCollection and AcceptAsyncSafe is called.\nI have tried to describe all of the functions that I think merit a description but I have been involved in this sort of code for years now so if you have any queries feel free to just drop a comment and I will try to help.\nWhen looking through the code remember that this is just a demo, I am currently still working on a few things but may offer the full API available for download at a later date or put it on GitHub.\nIn part four we are going to compare some of the differences in operation between the xxxAsync and the IAsync pattern, obviously there is a lot more code and inherent complexity in this implementation but in high volume situations it makes a lot of difference.\nSee you next time.\n","date":"January 20, 2011","externalUrl":null,"permalink":"/programming/2011-01-20-sockets-and-bockets-part-3/","section":"Blog","summary":"Welcome to part three! # As promised heres a description of the inner workings. I’m sick to death of typing SocketAsyncEventArgs so from now on I will refer to it as SAEA.\n","title":"Sockets and Bockets 3","type":"programming"},{"content":" Welcome to part two # Lets jump in at the deep end and take a look at some code\u0026hellip;\nWhen you look at the method syntax for the xxxAsync methods you will notice they return a boolean value that indicates if the method completed synchronously, this means that you have to check the return value every time you use one of the methods and invoke the callback yourself if it completes synchronously. In practice this hardly ever happens, and normally only on a send operation. But as it is a possibility we will add module with a some extension methods in to help us out, this will make the code more readable and avoid unnecessary duplication.\nSocketExtensions # module SocketExtensions open System open System.Net open System.Net.Sockets type Socket with /// extension method to make async based call easier, this ensures the callback always gets /// called even if there is an error or the async method completed syncronously member s.InvokeAsyncMethod( asyncmethod, callback, args:SocketAsyncEventArgs) = let result = asyncmethod args if result \u0026lt;\u0026gt; true then callback args member s.AcceptAsyncSafe(callback, args) = s.InvokeAsyncMethod(s.AcceptAsync, callback, args) member s.ReceiveAsyncSafe(callback, args) = s.InvokeAsyncMethod(s.ReceiveAsync, callback, args) member s.SendAsyncSafe(callback, args) = s.InvokeAsyncMethod(s.SendAsync, callback, args) member s.DisconnectAsyncSafe(callback, args) = s.InvokeAsyncMethod(s.DisconnectAsync, callback, args) Now lets get down to business, the next few types have a fair bit of code in them so I will briefly explain each type in turn:\nBocketPool # A BocketPool is a combination of a [SocketAsyncEventArgs](http://msdn.microsoft.com/en- us/library/system.net.sockets.socketasynceventargs.aspx) object and a chunk of memory allocated in an array. The array is sliced up into sections and allocated for each send or receive operation by setting a start and end index using SetBuffer(). If you remember last time I mentioned that a lot of memory fragmentation can occur during sending and receiving due to continuously allocating memory buffers on the Socket object, this is primarily done through the BeginSend and BeginReceive methods passing in a byte array. Using the BocketPool it a great way of reducing the amount of garbage collection during heavy traffic.\nThe other major difference with SocketAsyncEventArgs is the way in which you make the send and receive calls, heres the general flow that occurs:\nCreate a SocketAsyncEventArgs object or get one from a pool. Allocate an array to the buffer. Allocate an offset and length to the buffer. Allocate a callback method. Call Socket.xxxAsync passing in the SocketAsyncEventArgs, the operation will complete and invoke the callback. What we are going to do is wrap the whole creation, array allocation, and offsetting to the BocketPool:\nnamespace Fes open System open System.Net.Sockets open System.Collections.Concurrent type BocketPool( number, size, callback) as this = let number = number let size = size let totalsize = (number * size) let buffer = Array.create totalsize 0uy let pool = new BlockingCollection\u0026lt;SocketAsyncEventArgs\u0026gt;(number:int) let mutable disposed = false let cleanUp() = if not disposed then disposed \u0026lt;- true pool.CompleteAdding() while pool.Count \u0026gt; 1 do (pool.Take() :\u0026gt; IDisposable).Dispose() pool.Dispose() do let rec loop n = match n with | x when x \u0026lt; totalsize -\u0026gt; let saea = new SocketAsyncEventArgs() saea.Completed |\u0026gt; Observable.add( fun saea -\u0026gt; (callback saea)) saea.SetBuffer(buffer, n, size) this.CheckIn(saea) loop (n + size) | _ -\u0026gt; () loop 0 member this.CheckOut()= pool.Take() member this.CheckIn(saea)= pool.Add(saea) member this.Count = pool.Count interface IDisposable with member this.Dispose() = cleanUp() Next up we have to look at the Connection and the Tcplistener types as two interconnected entities:\nThe TcpListener listens for a connection on a socket and port number. The client connects to the server. An accept socket is allocated to the client, at this point we have one socket for the server and once for each client. We also need to allocate a BocketPool for send and receive operation for each client To simplify things we are going to encapsulate the accept socket management into a type, it will also need a corresponding BocketPool to service any send and receive operations to and from the client Connection # namespace Fes open System open System.Net open System.Net.Sockets open System.Collections.Generic open System.Collections.Concurrent open System.Threading open SocketExtensions type Connection(maxreceives, maxsends, size, socket:Socket) as this = let socket = socket let maxreceives = maxreceives let maxsends = maxsends let sendPool = new BocketPool(maxsends, size, this.sendCompleted ) let receivePool = new BocketPool(maxreceives, size, this.receiveCompleted) let mutable disposed = false let mutable anyErrors = false let cleanUp() = if not disposed then disposed \u0026lt;- true socket.Shutdown(SocketShutdown.Both) socket.Disconnect(false) socket.Close() (sendPool :\u0026gt; IDisposable).Dispose() (receivePool :\u0026gt; IDisposable).Dispose() member this.Start() = socket.ReceiveAsyncSafe(this.receiveCompleted, receivePool.CheckOut()) member this.Stop() = socket.Close(2) member this.receiveCompleted (args: SocketAsyncEventArgs) = try match args.LastOperation with | SocketAsyncOperation.Receive -\u0026gt; match args.SocketError with | SocketError.Success -\u0026gt; socket.ReceiveAsyncSafe( this.receiveCompleted, receivePool.CheckOut()) let data = Array.create args.BytesTransferred 0uy Buffer.BlockCopy(args.Buffer, args.Offset, data, 0, data.Length) let client = args.RemoteEndPoint args.RemoteEndPoint \u0026lt;- null data |\u0026gt; printfn \u0026#34;received data: %A\u0026#34; | _ -\u0026gt; args.SocketError.ToString() |\u0026gt; printfn \u0026#34;socket error on receive: %s\u0026#34; | _ -\u0026gt; failwith \u0026#34;unknown operation, should be receive\u0026#34; finally receivePool.CheckIn(args) member this.sendCompleted (args: SocketAsyncEventArgs) = try match args.LastOperation with | SocketAsyncOperation.Send -\u0026gt; match args.SocketError with | SocketError.Success -\u0026gt; () | SocketError.NoBufferSpaceAvailable | SocketError.IOPending | SocketError.WouldBlock -\u0026gt; if not(anyErrors) then anyErrors \u0026lt;- true failwith \u0026#34;Buffer overflow or send buffer timeout\u0026#34; | _ -\u0026gt; args.SocketError.ToString() |\u0026gt; printfn \u0026#34;socket error on send: %s\u0026#34; | _ -\u0026gt; failwith \u0026#34;invalid operation, should be receive\u0026#34; finally sendPool.CheckIn(args) member this.Send (msg:byte[]) = let s = sendPool.CheckOut() Buffer.BlockCopy(msg, 0, s.Buffer, s.Offset, msg.Length) socket.SendAsyncSafe(this.sendCompleted, s) Finally here\u0026rsquo;s the TcpListener type. It is responsible for creating an initial Connection object for each client and starts asynchronous sending messages to that client once a second, also notice that there is another BlockingCollection involved, this is somewhat simpler than the usage in the bocketPool as we have no buffer to manage here.\nIt is possible to fill the initial Buffer property, this causes the buffer to be sent to the client as soon as it has connected to the server, this can be useful to sent initial data to the client, such as protocol definitions etc) A finite number of connections can occur before blocking will occur depending on the number of AsyncEventArgs in the collection, this stops potential denial of service attacks due to too many connection being made. TcpListener # namespace Fes open System open System.Net open System.Net.Sockets open System.Collections.Generic open System.Collections.Concurrent open System.Threading open SocketExtensions type TcpListener(maxaccepts, maxsends, maxreceives, size, port, backlog) as this = let createTcpSocket() = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp) let createListener (ip:IPAddress, port, backlog) = let s = createTcpSocket() s.Bind(new IPEndPoint(ip, port)) s.Listen(backlog); s let listeningSocket = createListener( IPAddress.Loopback, port, backlog) let initPool (maxinpool, callback) = let pool = new BlockingCollection\u0026lt;SocketAsyncEventArgs\u0026gt;(maxinpool:int) let rec loop n = match n with | x when x \u0026lt; maxinpool -\u0026gt; let saea = new SocketAsyncEventArgs() saea.Completed |\u0026gt; Observable.add callback pool.Add saea loop (n+1) | _ -\u0026gt; () loop 0 pool let acceptPool = initPool (maxaccepts, this.acceptcompleted) let newConnection socket = new Connection (maxreceives, maxsends, size, socket) let testMessage = Array.init\u0026lt;byte\u0026gt; 128 (fun _ -\u0026gt; 1uy) let header = Array.init\u0026lt;byte\u0026gt; 1 (fun _ -\u0026gt; 1uy) let mutable disposed = false //mutable state from original let mutable anyErrors = false let mutable requestCount = 0 let mutable numWritten = 0 //async code from original let asyncWriteStockQuote(connection:Connection) = async { do! Async.Sleep 1000 connection.Send(testMessage) Interlocked.Increment(\u0026amp;numWritten) |\u0026gt; ignore } //async code from original let asyncServiceClient (client: Connection) = async { client.Send(header) while true do do! asyncWriteStockQuote(client) } let startSending connection = Async.Start (async { try use _holder = connection do! asyncServiceClient connection with e -\u0026gt; if not(anyErrors) then anyErrors \u0026lt;- true Console.WriteLine(\u0026#34;server ERROR\u0026#34;) raise e } ) let reportConnections = Interlocked.Increment(\u0026amp;requestCount) |\u0026gt; ignore if requestCount % 1000 = 0 then requestCount |\u0026gt; printfn \u0026#34;%A Clients accepted\u0026#34; let cleanUp() = if not disposed then disposed \u0026lt;- true listeningSocket.Shutdown(SocketShutdown.Both) listeningSocket.Disconnect(false) listeningSocket.Close() member this.acceptcompleted (args : SocketAsyncEventArgs) = try match args.LastOperation with | SocketAsyncOperation.Accept -\u0026gt; match args.SocketError with | SocketError.Success -\u0026gt; listeningSocket.AcceptAsyncSafe( this.acceptcompleted, acceptPool.Take()) //create new connection let connection = newConnection args.AcceptSocket connection.Start() //update stats reportConnections //async start of messages to client startSending connection //remove the AcceptSocket because we will be reusing args args.AcceptSocket \u0026lt;- null | _ -\u0026gt; args.SocketError.ToString() |\u0026gt; printfn \u0026#34;socket error on accept: %s\u0026#34; | _ -\u0026gt; args.LastOperation |\u0026gt; failwith \u0026#34;Unknown operation, should be accept but was %a\u0026#34; finally acceptPool.Add(args) member this.start () = listeningSocket.AcceptAsyncSafe( this.acceptcompleted, acceptPool.Take()) while true do Thread.Sleep 1000 let count = Interlocked.Exchange(\u0026amp;numWritten, 0) count |\u0026gt; printfn \u0026#34;Quotes per sec: %A\u0026#34; member this.Close() = cleanUp() interface IDisposable with member this.Dispose() = cleanUp() Its a fair bit of code to take in at once, so Ill leave you with it to ponder over. Ill be explaining all of the interesting bits in more detail in part three\u0026hellip;\nPlease feel free to leave any comments you have, especially on better use of functional constructs.\n","date":"January 14, 2011","externalUrl":null,"permalink":"/programming/2011-01-14-sockets-and-bockets-part-2/","section":"Blog","summary":"Welcome to part two # Lets jump in at the deep end and take a look at some code…\nWhen you look at the method syntax for the xxxAsync methods you will notice they return a boolean value that indicates if the method completed synchronously, this means that you have to check the return value every time you use one of the methods and invoke the callback yourself if it completes synchronously. In practice this hardly ever happens, and normally only on a send operation. But as it is a possibility we will add module with a some extension methods in to help us out, this will make the code more readable and avoid unnecessary duplication.\n","title":"Sockets and Bockets 2","type":"programming"},{"content":" Welcome to part 1 # A while back I read an interesting article by Brian McNamara f-async-on-the-server-side which describes C# and F# versions of a simple asynchronous socket server, one of the driving forces behind the article was how F# can wrap the traditional asynchronous model with Asynchronous Workflows, this produces nice clean simple code compared to the C# version which uses lambda expressions, the code looks quite ugly in this style! However thats not the end of the story, a lot of memory fragmentation can occur using the APM model when there is a high throughput, so I thought I would see if I could take this a step further\u0026hellip;\nThere are some lesser known methods that were added to the Socket class in .Net 2.0 SP1: ReceiveAsync, SendAsync, ConnectAsync and DisconnectAsync. These methods use an event driven model and do not result in the creation of AsyncResult objects, these are created on every asynchronous call in the traditional Socket Begin/End methods. Once you have thousands of clients sending and receiving thousands of messages all of the object creation can really have an adverse effect on performance on the garbage collected, you will regularly see the AsyncResult objects hitting Generation 1 and 2.\nTo use the xxxAsync methods you have pass a SocketAsyncEventArgsobject which is assigned callback method and a buffer, the callback method called asynchronously when the operation completes and is passed the corresponding SocketAsyncEventArgs object, this allows you query the buffer in a receive operation.\nThe scope of this series of articles is to initially replicate Brian\u0026rsquo;s demo using F# and a pool of SocketAsyncEventArgs and a contiguous block of memory to hold the data being sent and received on the Socket, this again further reduces memory fragmentation on the send and receive buffers.\nI have successfully developed an enterprise server for a client using this method, it processed thousands of simultaneous connected clients and messages, key components in the system were the High performance sockets, a pipeline processor and a highly efficiency means of data compaction, I will only be including the High performance sockets in this series but the other components will be at a later date in separate articles. Interestingly all of the code was originally developed in c# but had a distinctly functional style, even the Pipeline Processing is reminiscent of functional composition using the F# pipeline operator |\u0026gt; although an analogue of attach and detach was used which in itself is declarative.\nAlthough there is no code in this article there is plenty in the next!\nPlease feel free to leave comments or add any suggestions, hope to see you next time\u0026hellip;\n","date":"January 13, 2011","externalUrl":null,"permalink":"/programming/2011-01-13-sockets-and-bockets-1/","section":"Blog","summary":"Welcome to part 1 # A while back I read an interesting article by Brian McNamara f-async-on-the-server-side which describes C# and F# versions of a simple asynchronous socket server, one of the driving forces behind the article was how F# can wrap the traditional asynchronous model with Asynchronous Workflows, this produces nice clean simple code compared to the C# version which uses lambda expressions, the code looks quite ugly in this style! However thats not the end of the story, a lot of memory fragmentation can occur using the APM model when there is a high throughput, so I thought I would see if I could take this a step further…\n","title":"Sockets and Bockets 1","type":"programming"}]