My two cents about software development on the web

Cloud Platforms

Warning: file_put_contents(C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache/0bac12a71080e2f0238f1b173488ab6f.spc) [function.file-put-contents]: failed to open stream: Permission denied in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 8680

Warning: C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache is not writeable. Make sure you've set the correct relative or absolute path, and that the location is server-writable. in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 1781

Warning: file_put_contents(C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache/ad31bb8bf02bbf58903d4663ca4ee574.spc) [function.file-put-contents]: failed to open stream: Permission denied in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 8680

Warning: C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache is not writeable. Make sure you've set the correct relative or absolute path, and that the location is server-writable. in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 1781
  • Windows Server 2012 R2, IIS 8.5, WebSockets and .NET 4.5.2 (AppHarbor Blog)

    During the last couple of weeks we've upgraded worker servers in the US and EU regions to support Windows Server 2012 R2, IIS 8.5 and .NET 4.5.2. Major upgrades like this can be a risky and lead to compatibility issues, and the upgrade was carefully planned and executed to maximize compatibility with running applications. Application performance and error rates have been closely monitored throughout the process and fortunately, chances are you haven't noticed a thing: We've detected migration-related issues with less than 0.1% of running applications.

    Many of the new features and configuration improvements enabled by this upgrade will be gradually introduced over the coming months. This way we can ensure a continued painless migration and maintain compatibility with the previous Windows Server 2008 R2/IIS 7.5 setup, while we iron out any unexpected kinks if and when they crop up. A few changes have however already been deployed that we wanted to fill you in on.

    WebSocket support and the beta region

    Last year the beta region featuring experimental WS2012 and WebSockets support was introduced. The beta region allowed customers to test existing and new apps on the new setup while we prepared and optimized it for production use. This approach has been an important factor in learning about subtle differences between the server versions, and addressing pretty much all compatibility issues before upgrading the production regions. Thanks to all the customers who provided valuable feedback during the beta and helped ensure a smoother transition for everyone.

    An important reason for the server upgrade was to support WebSocket connections. Now that the worker servers are running WS2012 and IIS 8.5 we've started doing just that. Applications in the old beta region have been merged into the production US region and the beta region is no longer available when you create a new application.

    Most load balancers already support WebSockets and the upgrade is currently being rolled out to remaining load balancers. Apps created since August 14th fully support WebSockets and no configuration is necessary: AppHarbor will simply detect and proxy connections as expected when a client requests a Connection: Upgrade.

    Some libraries, such as SignalR, will automatically detect and prefer WebSocket connections when supported by both the server and client. Until WebSocket connections are supported on all load balancers some apps may attempt and fail during the WebSocket handshake. This should not cause issues since these libraries will fall back to other supported transports, and affected apps will automatically be WebSocket-enabled when supported by the load balancers.

    CPU throttling

    One of the major challenges that has held back this upgrade is a change in the way we throttle worker CPU usage. CPU limitations are the same as before, but the change can affect how certain CPU-intensive tasks are executed. Resources and documentation on this subject are limited, but testing shows that CPU time is more evenly scheduled across threads, leading to higher concurrency, consistency and stability within processes. While this is overall an improvement it can also affect peak performance on individual threads, and we're currently investigating various approaches to better support workloads affected by this.

    For the curious, we previously used a CPU rate limit registry setting to limit CPU usage per user account, but this is no longer supported on Windows Server 2012. We now use a combination of IIS 8's built-in CPU throttling and a new CPU rate control for job objects to throttle background workers.

    If you've experienced any issues with this upgrade or have feedback about the process, please don't hesitate to reach out.

  • Heartbleed Security Update (AppHarbor Blog)

    Updated on April 10, 2014 with further precautionary steps in the "What can you do" section below.

    On April 7, 2014, a serious vulnerability in the OpenSSL library (CVE-2014-0160) was publicly disclosed. OpenSSL is a cryptography library used for the majority of private communications across the internet.

    The vulnerability, nicknamed "Heartbleed", would allow an attacker to steal secret certificates keys, names and passwords of users and other secrets encrypted using the OpenSSL library. As such it represents a major risk for a large number of internet application and services, including AppHarbor.

    What has AppHarbor done about this

    AppHarbor responded to the announcement by immediately taking steps to remediate the vulnerability:

    1. We updated all affected components with the updated, secure version of OpenSSL within the first few hours of the announcement. This included SSL endpoints and load balancers, as well as other infrastructure components used internally at AppHarbor.
    2. We re-keyed and redeployed all potentially affected AppHarbor SSL certificates (including the piggyback * certificate), and the old certificates are being revoked.
    3. We notified customers with custom SSL certificates last night, so they could take steps to re-key and reissue certificates, and have the old ones revoked.
    4. We reset internal credentials and passwords.
    5. User session cookies were revoked, requiring all users to sign in again.

    Furthermore, AppHarbor validates session cookies against your previously known IP addresses as part of the authorization process. This has reduced the risk of a stolen session cookie being abused. Perfect forward secrecy was deployed to some load balancers, making it impossible to read intercepted and encrypted communication with stolen keys. Forward secrecy has since been deployed to all load balancers hosted by AppHarbor.

    What can you do

    We have found no indication that the vulnerability was used to attack AppHarbor. By quickly responding to the issue and taking the steps mentioned above we effectively stopped any further risk of exposure. However, due to the nature of this bug, we recommend users who want to be extra cautious to take the following steps:

    1. Reset your AppHarbor password.
    2. Review the sign-in and activity history on your user page for any suspicious activity.
    3. Revoke authorizations for external applications that integrates with AppHarbor.
    4. Recreate, reissue and reinstall custom SSL certificates you may have installed, and revoke the old ones. Doing this may revoke the old certificates, so make sure you're ready to install the new certificates.
    5. Read the details about the Heartbleed bug here and assess the risks relative to your content.

    Updated instructions (April 10, 2014):

    While we still have not seen any abuse on AppHarbor as a result of this bug, we now also encourage you to take these precautionary steps:

    1. Reset your build URL token.
    2. If you're using one of the SQL Server or MySQL add-ons: Reset the database password. Go to the add-on's admin page and click the "Reset Password" button. This will immediately update the configuration on AppHarbor and redeploy the application (with a short period of downtime until it is redeployed).
    3. If you're using the Memcacher add-on: Reinstall the add-on by uninstalling and installing it.
    4. Rotate/update sensitive information in your own configuration variables.

    If you have hardcoded passwords/connection strings for any your add-ons this is a good opportunity to start using the injected configuration variables. You can find instructions for the SQL add-ons here and the Memcacher add-on here. This way your application is automatically updated when you reset the add-ons, or when an add-on provider updates the configuration. If this is not an option you should immediately update your code/configuration files and redeploy the application after the configuration is updated.

    Stay tuned

    Protecting your code and data is our top priority, and we continue to remediate and asses the risks in response to this issue. We'll keep you posted with any new developments, so stay tuned on Twitter and the blog for important updates. We're of course also standing by on the support forums if you have any questions or concerns.

  • Librato integration and built-in perfomance metrics (AppHarbor Blog)

    Librato Dashboard

    Being able to monitor and analyze key application metrics is an essential part of developing stable, performant and high quality web services that meets you business requirements. Today we’re announcing a great new set of features to provide a turnkey solution for visualizing, analyzing and acting on key performance metrics. On top of that we’re enabling you to easily track your own operational metrics. In this blog post we’ll look at how the pieces tie together.

    Librato integration

    The best part of today’s release is our new integration with Librato for monitoring and analyzing metrics. Librato is an awesome and incredibly useful service that enables you to easily visualize and correlate metrics, including the new log-based performance metrics provided by AppHarbor (described in more details below).

    Librato Dashboard

    Librato is now available as an add-on and integrates seamlessly with your AppHarbor logs. When you provision the add-on, Librato will setup a preconfigured dashboard tailored for displaying AppHarbor performance data and you can access it immediately by going to the Librato admin page. Everything will work out of the box without any further configuration and your logs will automatically be sent to Librato using a log drain.

    When log messages containing metric data are sent to Librato they’re transformed by an l2met service before being sent to their regular API. A very cool feature of the l2met service is that it can automatically calculate some useful metrics. For instance, it’ll calculate the median response time as well as the the 99th and 95th percentile of measurements such as response times. The perc99 response time means the response time of the 99% fastest responses. It can be useful to know this value since it's less affected by a few very slow responses than the average. Among other things this provides a good measurement of the browsing experience for most of your users.

    Librato Dashboard

    The l2met project was started by Ryan Smith - a big shout-out and thanks to him and the Librato team for developing this great tool.

    For more information about how to integrate with Librato and details about the service please refer to the documentation here. Also check out their announcement blog post about the integration.

    Built-in performance metrics

    AppHarbor can now write key runtime performance metrics directly to your application’s log stream as l2met 2.0 formatted messages similar to this:

    source=web.5 sample#memory.private_bytes=701091840
    source=web.5 sample#process.handles=2597
    source=web.5 sample#cpu.load_average=1.97

    These are the messages Librato uses as well and most of them are written every 20 seconds. They allow for real-time monitoring of worker-specific runtime metrics such as CPU (load average) and memory usage, as well as measurements of response time and size reported from the load balancers. Because these metrics are logged to your log stream you can also consume them in the same way you’d usually view or integrate with your logs.

    Load average run-time metrics

    Performance data collection takes place completely out-of-process, without using a profiler, and it can be enabled and disabled without redeploying the application. This means that monitoring won’t impact application performance at all and that a profiler (such as New Relic) can still be attached to the application.

    Writing custom metrics

    The performance data provided by AppHarbor is probably not the only metrics you want to track. You can of course integrate directly with Librato’s API, but the l2met integration makes it easier than ever to track your own metrics, and the paid Librato plans includes the ability to track custom metrics exactly for that purpose.

    You can start writing your own metrics simply by sending an l2met-formatted string to your logs. Last week we introduced the Trace Logging feature which is perfect for this, so writing your custom metrics can now be done with a simple trace:


    To make this even easier we’ve built the metric-reporter library (a .NET port of Librato’s log-reporter) to provide an easy to use interface for writing metrics to your log stream. You can install it with NuGet:

    Install-Package MetricReporter

    Then initialize a MetricReporter which writes to a text writer:

    var writer = new L2MetWriter(new TraceTextWriter);
    var reporter = new MetricReporter(metricWriter);

    And start tracking your own custom metrics:

    reporter.Measure("payload.size", 21276);
    reporter.Measure("twitter.lookup.time", () =>
        //Do work

    On Librato you can then view charts with these new metrics along with the performance metrics provided by AppHarbor, and add them to your dashboards, aggregate and correlate data, set up alerts etc. The MetricReporter library will take care of writing l2met-formatted metrics using the appropriate metric types and write to the trace or another IO stream. Make sure to inspect the README for more examples and information on configuration and usage.

    That’s all we have for today. There’ll be more examples on how you can use these new features soon, but for now we encourage you to take it for a spin, install the Librato add-on and test the waters for yourself. We’d love to hear what you think so if there are other metrics you’d like to see or if you experience any issues please hit us up through the usual channels.

  • Introducing Trace Logging (AppHarbor Blog)

    Today we’re happy to introduce trace message integration with your application log. With tracing you can very easily log trace messages to your application's log stream by using the built-in tracing capabilities of the .NET framework from anywhere in your application.

    When introducing the realtime logging module a while back we opened up access to collated log data from load balancers, the build and deploy infrastructure, background workers and more. Notably missing however was the ability to log from web workers. We’re closing that gap with tracing, which can be used in both background and web workers.

    How to use it

    The trace feature integrates with standard .NET tracing, so you don’t have to make any changes to your application to use it. You can simply log traces from your workers with the System.Diagnostics.Trace class:

    Trace.TraceInformation("Hello world");

    This will yield a log message containing a timestamp and the source of the trace in your application’s log like so:

    2014-01-22T06:46:48.086+00:00 app web.1 Hello World

    You can also use a TraceSource by specifying the trace source name AppHarborTraceSource:

    var traceSource = new TraceSource("AppHarborTraceSource", defaultLevel: SourceLevels.All);
    traceSource.TraceEvent(TraceEventType.Critical, 0, "Foo");

    You may not always want noisy trace messages in your logs and you can configure the trace level on the "Logging" page. There are 4 levels: All, Warning, Error and None. Setting the trace level will update the configuration without redeploying or restarting the application. This is often desirable if you need to turn on tracing when debugging and diagnosing an ongoing or state-related issue.

    Configure Trace level

    There are a number of other ways to use the new tracing feature including:

    • ASP.NET health monitoring (for logging exceptions, application lifecycle events etc).
    • A logging library such as NLog (Trace target) or log4net (TraceAppender).
    • Integrating with ETW (Event Tracing for Windows) directly using the injected event provider id.

    Anything that integrates with .NET tracing or ETW should work, and you can find more details and examples in this knowledge base article.

    All new applications have tracing enabled by default. Tracing can be enabled for existing applications on the "Logging" page.

    How does it work

    Under the hood we’re using ETW for delivering log messages to the components that are responsible for sending traces to your log stream. Application performance is unaffected by the delivery of log messages as this takes place completely out of process. Note however that messages are buffered for about a second and that some messages may be dropped if you’re writing excessively to the trace output.

    When tracing is enabled, AppHarbor configures your application with an EventProviderTraceListener as a default trace listener. While you can integrate directly with ETW as well we recommend using the Trace or TraceSource approaches described above.

    Viewing trace messages

    Traces are collated with other logging sources in your log stream, so you can consume them in the same way you’re used to. You can view log messages using the command line interface, the web viewer or set up a log drain to any HTTP, HTTPS or syslog endpoint. For more information about the various integration points please refer to this article.

    Viewing trace messages in console

    We’ve got a couple of cool features that builds on this ready soon, so stay tuned and happy tracing!

  • .NET 4.5.1 is ready (AppHarbor Blog)

    Microsoft released .NET 4.5.1 a while back, bringing a bunch of performance improvements and new features to the framework. Check out the announcement for the details.

    Over the past few weeks we have updated our build infrastructure and application servers to support this release. We're happy to report that AppHarbor now supports building, testing and running applications targeting the .NET 4.5.1 framework, as well as solutions created with Visual Studio 2013 and ASP.NET MVC 5 applications.

    There are no known issues related to this release. If you encounter problems, please refer to the usual support channels and we'll help you out.

    .NET logo

  • Integrated NuGet Package Restore (AppHarbor Blog)

    A few months ago the NuGet team released NuGet 2.7, which introduced a new approach to package restore. We recently updated the AppHarbor build process to adopt this approach and integrate the new NuGet restore command. AppHarbor will now automatically invoke package restore before building your solution.

    Automatically restoring packages is a recommended practice, especially because you don’t have to commit the packages to your repository and can keep the footprint small. Until now we’ve recommended using the approach desribed in this blog post to restore NuGet packages when building your application. This has worked relatively well, but it’s also a bit of a hack and has a few caveats:

    • Some NuGet packages rely files that needs to be present and imported when MSBuild is invoked. This has most notably been an issue for applications relying on the Microsoft.Bcl.Build package for the reasons outlined in this article.
    • NuGet.exe has to be committed and maintained with the repository and project and solution files needs to be configured.
    • Package restore can intermittently fail in some cases when multiple projects are built concurrently.

    With this release we expect to eliminate these issues and provide a more stable, efficient and streamlined way of handling package restore.

    If necessary, NuGet can be configured by adding a NuGet.config file in the same directory as your solution file (or alternatively in a .nuget folder under your solution directory). You usually don't have to configure anything if you’re only using the official NuGet feed, but you’ll need to configure your application if it relies on other package sources. You can find an example configuration file which adds a private package source in the knowledge base article about package restore and further documentation for NuGet configuration files can be found here.

    If you hit any snags we’re always happy to help on our support forums.

    NuGet logo

  • New Relic Improves Service and Reduces Price (AppHarbor Blog)

    New Relic

    We're happy to announce that New Relic has dropped the price of the Professional add-on plan from $45/month to $19/month per worker unit. Over the years New Relic has proven to be a really useful tool for many of our customers, and we're pleased that this price drop will make the features of New Relic Professional more accessible to everyone using AppHarbor.

    Highlights of the Professional plan include:

    • Unlimited data retention
    • Real User Monitoring (RUM) and browser transaction tracing
    • Application transaction tracing, including Key Transactions and Cross Application Tracing
    • Advanced SQL and slow SQL analysis

    You can find more information about the benefits of New Relic Pro on the New Relic website (

    Service update

    The New Relic agent was recently upgraded to a newer version which brings support for some recently introduced features as well as a bunch of bug fixes. Time spent in the request queue is now reported and exposed directly in the New Relic interface. Requests are rarely queued for longer than a few milliseconds, but it can happen if your workers are under load. When more time is spent in the request queue it may be an indicator that you need to scale your application to handle the load efficiently.

    We're also making a few changes to the way the New Relic profiler is initialized with your applications. This is particularly relevant if you've subscribed to New Relic directly rather than installing the add-on through AppHarbor. Going forward you'll need to add a NewRelic.LicenseKey configuration variable to make sure the profiler is attached to your application. We recommend that you make this change as soon as possible. If you're subscribed to the add-on through AppHarbor no action is required and the service will continue to work as usual.

  • Found Elasticsearch add-on available (AppHarbor Blog)

    Found ElasticSearch

    Found provides fully hosted and managed Elasticsearch clusters; each cluster has reserved memory and storage ensuring predictable performance. The HTTPS API is developer-friendly and existing Elasticsearch libraries such as NEST, Tire, PyES and others work out of the box. The Elasticsearch API is unmodified, so for those with an existing Elasticsearch integration it is easy to get started.

    For production and mission critical environments customers can opt for replication and automatic failover to a secondary site, protecting the cluster against unplanned downtime. Security has a strong focus: communication to and from the service is securely transmitted over HTTPS (SSL) and data is stored behind multiple firewalls and proxies. Clusters run in isolated containers (LXC) and customisable ACLs allow for restricting access to trusted people and hosts.

    In the event of a datacenter failure, search clusters are automatically failed over to a working datacenter or, in case of a catastrophic event, completely rebuilt from backup.

    Co-founder Alex Brasetvik says: "Found provides a solution for companies who are keen to use Elasticsearch but not overly keen to spend their time and money on herding servers! We provide our customers with complete cluster control: they can scale their clusters up or down at any time, according to their immediate needs. It's effortless and there's zero downtime."

    More information and price plans are available on the add-on page.

  • Introducing Realtime Logging (AppHarbor Blog)

    Today we're incredibly excited to announce the public beta of our brand new logging module. Starting immediately all new applications created on AppHarbor will have logging enabled. You can enable it for your existing apps on the new "Logging" page.

    We know all too well that running applications on a PaaS like AppHarbor sometimes can feel like a black box. So far we haven't had a unified, simple and efficient way to collate, present and distribute log events from the platform and your apps.

    That's exactly what we wanted to address with our logging solution, and based on the amazing feedback from private beta users we feel confident that you'll find it useful for getting insight about your application and AppHarbor. A big thanks to all the beta testers who have helped us refine and test these new features.

    The new logging module collates log messages from multiple sources, including almost all AppHarbor infrastructure component and your applications - API changes, load balancer request logs, build, deploy and stdout/stderr from your background workers and more can now be accessed and sent to external services in real time.

    Captain's log Consider yourself lucky we're not that much into skeuomorphism


    We're providing two interfaces "out of the box" - a convenient web-interface can be accessed on the Logging page and a new log command has been added to the CLI. [Get the installer directly from here or install with Chocolatey cinst appharborcli.install. To start a "tailing" log session with the CLI, you can for instance run appharbor log -t -s appharbor. Type appharbor log -h to see all options. Log web interface

    The web interface works a bit differently, but try it out and let us know what you think - it's heavily inspired by the project who have built a great client side interface for viewing, filtering, searching and splitting logs into multiple "screens".

    Log web interface


    One of the most useful and interesting aspects of today's release is the flexible integration points it provides. Providing access to your logs in realtime is one thing, but AppHarbor will only store the last 1500 log messages for your application. Storing, searching, viewing and indexing logs can be fairly complex and luckily many services already exists that helps you make more sense of your log data.

    We've worked with Logentries to provide a completely automated and convenient way for sending AppHarbor logs to them when you add their add-on. When you add the Logentries add-on your application can automatically be configured to send logs to Logentries, and Logentries will be configured to display log messages in AppHarbor's format.

    Logentries integration

    You can also configure any syslog (TCP), HTTP and HTTPS endpoint you like with log "drains". You can use this to integrate with services like Loggly and Splunk, or even your own syslog server or HTTP service. More details about log drains are available in the this knowledge base article and the drain API documentation.

    Finally there's a new new Log session API endpoint that you can use to create sessions similar to the ones used by the interfaces we provide.


    If you've ever used Heroku you'll find most of these features very familiar. That's no coincidence - the backend is based on Heroku's awesome distributed syslog router, Logplex. Integrating with Logplex makes it a lot easier for add-on providers who already support Heroku's Logplex to integrate with AppHarbor, while giving us a scalable and proven logging backend to support thousands of deployed apps.

    Logplex is also in rapid, active development, and a big shout-out to the awesome people at Heroku who are building this incredibly elegant solution. If you're interested in learning more about Logplex we encourage you to check out the project on Github and try it for yourself. We've built a client library for interacting with Logplex's HTTP API and HTTP log endpoints from .NET apps - let us know if you'd like to use this and we'll be happy to open source the code. The Logplex documentation on stream management is also useful for a high-level overview of how Logplex works.

    Next steps

    With this release we've greatly improved the logging experience for our customers. We're releasing this public beta since we know it'll be useful to many of you as it is, but we're by no means finished. We want to add even more log sources, provide more information from the various infrastructure components and integrate with more add-on providers. Also note that request logs are currently only available on shared load balancers, but it will be rolled out to all load balancers soon. If you find yourself wanting some log data that is not currently available please let us know. We now have a solid foundation to provide you with the information you need when you need it, and we couldn't be more excited about that.

    We'll provide you with some examples and more documentation for these new features over the next couple of weeks, but for now we hope you'll take it for a spin and test the waters for yourself. Have fun!

  • Introducing PageSpeed optimizations (AppHarbor Blog)

    Today we've introducing a new experimental feature: Google PageSpeed optimizations support. The PageSpeed module is a suite of tools that tries to optimize web page latency and bandwidth usage of your websites by rewriting your content to implement web performance best practices. Reducing the number of requests to a single domain, optimizing cache policies and compressing content can significantly improve web performance and lead to a better user experience.

    With PageSpeed optimization filters we're making it easier to apply some of these best practices, and provide a solution that efficiently and effortlessly speed up your web apps. The optimizations takes place at the load balancer level and works for all web applications no matter what framework or language you use.

    As an example of how this works you can inspect the HTML and resources of this blog to see some of the optimizations that are applied. Analyzing with the online PageSpeed insights tool yields a "PageSpeed score" of 88 when enabled versus 73 when disabled. Not too bad considering it only took a click to enable it.

    PageSpeed button

    You can enable PageSpeed optimizations for your web application on the new "Labs" page, which can be found in the application navigation bar. The application will be configured with PageSpeed's core set of filters within a few seconds. We will then, among other things, apply these filters to your content:

    When you've enabled PageSpeed we recommend that you test the application to make sure it doesn't break anything. You can also inspect the returned content in your browser and if you hit any snags simply disable PageSpeed and let support know about it. Note that only content transferred over HTTP from your domain will be processed by PageSpeed filters. To optimize HTTPS traffic you can enable SPDY support (although that is currently only enabled on dedicated load balancers and in the beta region).

    We'll make more filters available later on, but for the beta we're starting out with a curated set of core filters, which are considered safe for most web applications. There are a few other cool filters we'll add support for later on - such as automatic sprite image generation and lazy-loading of images. Let us know if there are any filters in the catalog you think we should support!

  • Heroku CI Is Now Generally Available: Fast, Low Setup CI That’s Easy to Use (Heroku)
    18 May 2017 15:44

    Today we are proud to announce that Heroku CI, a low-configuration test runner for unit and browser testing that is tightly integrated with Heroku Pipelines, is now in General Availability.


    To build software with optimal feature release speed and quality, continuous integration (CI) is a popular and best practice, and is an essential part of a complete continuous delivery (CD) practice. As we have done for builds, deployments, and CD, Heroku CI dramatically improves the ease, experience, and function of CI. Now your energy can go into your apps, not your process.

    With today's addition of Heroku CI, Heroku now offers a complete CI/CD solution for developers in all of our officially supported languages: Node, Ruby, Java, Python, Go, Scala, PHP, and Clojure. As you would expect from Heroku, Heroku CI is simple, powerful, visual, and prescriptive. It is intended to provide the features and flexibility to be the complete CI solution for the vast majority of application development situations, serving use cases that range from small innovation teams, to large Enterprise projects.

    Easy to Setup and Use


    Configuration of Heroku CI is quite low (or none). There is no IT involved; Heroku CI is automatically available and coordinated for all apps in Heroku Pipelines. Just turn on Heroku CI for the Pipeline, and each push to GitHub will run your tests. Tests reside in the location that is the norm typical for each supported language, for example: test scripts in Go typically reside in the file named "function_test.go". These tests are executed automatically on each git push. So no learning curve is involved, and little reconfiguration is typically necessary when migrating to Heroku CI from Jenkins and other CI systems.

    For users who are also new to continuous delivery, we've made Heroku Pipelines set-up easier than ever with a straightforward 3-step setup that automatically creates and configures your review, development, staging, and production apps. All that's left is to click the "Tests" tab and turn on Heroku CI.

    Visual at Every Stage


    From setup, to running tests, to CI management, everything about Heroku CI is intended to be fully visual and intuitive -- even for users who are new to continuous integration. For each app, the status of the latest or currently running test run is shown clearly on the Pipelines page. Test actions are a click away, and fully available via the UI: re-run any test, run new tests against an arbitrary branch, search previous tests by branch or pull request, and see full detail for any previous test. And Heroku CI integrates seamlessly with GitHub - on every git push your tests run, allowing you to also see the test result within GitHub web or GitHub Desktop interfaces.

    CI users who want more granular control, direct debug access, and programmatic control of CI actions can use the CLI interface for Heroku CI.

    Power, Speed, and Flexibility

    For every test you run, Heroku CI creates and populates an ephemeral app environment that mirrors your Staging and Production environments. These CI apps are created automatically, and then destroyed immediately after test runs complete. All the add-ons, databases, and configurations your code requires are optimized for test speed, and parity with downstream environments. Over the beta period, we have been working with add-on partners to make sure the CI experience is fast and seamless.

    Setup and tear-down for each CI run happens in seconds. Because we use these ephemeral Heroku apps to run your tests, there is no queue time (as is common with many CI systems). Your tests run immediately, every time on dedicated Performance dynos.

    Across the thousands of participants in our public beta, most developers observed test runs completing significantly faster than expectations.


    We view CI as an essential part of effective development workflows, that is, part of good overall delivery process.

    Each CI-enabled Heroku Pipeline is charged just $10/month for an unlimited number of test runs. For each test run, dyno charges apply only for the duration of tests. We recommend and default to Performance-M dynos to power test runs, and you can specify other dyno sizes.

    Note that all charges are pro-rated per second, with no commitment, so you can try out Heroku CI for pennies -- usually with little modification to your existing test scripts.


    All Heroku Enterprise customers get unlimited CI-enabled Pipelines, and an unlimited number of test runs, all, of course, with zero queue time. No provisioning, authentication set-up, or management of CI is required for new projects, and Heroku CI can be turned on for any Heroku Pipeline with a single click.

    Existing Heroku Enterprise dyno credits are automatically used for test runs, and invoices will contain a new section listing the CI-enabled Pipelines alongside the account-wide dyno usage for CI test runs.

    All test run results are available at permanent URLs that can be referenced for compliance regimes, and all authentication is managed under existing Heroku Enterprise Teams (Org) security. Unification of security, authentication, billing between CI and production deployments, along with a prescriptive methodology across company projects, lets Enterprises innovate on Heroku with the agility of a start-up.

    Heroku-built, Community-hardened

    Some terms are not usually associated with CI systems: we think Heroku CI is among the most pleasant, beautiful software testing systems available -- and we have you to thank for this. More than 1500 beta users tested Heroku CI, surfacing bugs, offering suggestions; telling us that some webhooks got dropped, that an icon on the tab might be nice, that it should be more obvious how to re-run a test ... and roughly 600 other notes, many of which grew into e-mail conversations with you. As is the case with all software: we will still be perfecting. And we are pretty proud of what we have here. Thank you, and keep the comments coming!

    Get Started

    It's easy. Set-up a Heroku Pipeline and you're ready. There's even a two-minute video here and a simple how-to. Give it a spin, and let us know what you think.

  • The Future of Ember.js: An Interview With Tom Dale at EmberConf - Part Two (Heroku)
    11 May 2017 15:25

    This is the second of a two-part transcript from a recent interview with Tom Dale of Ember.js. In part one we discussed the history and direction of the Ember.js project. Continuing the discussion of the future for Ember.js, this post includes the rest of the interview, primarily focused on the Glimmer.js project. Some of the questions were omitted from these transcriptions for brevity, so we’re also releasing the nearly hour long audio file of the entire interview. Enjoy!

    Jonan: Let’s talk about Glimmer 2. If I understand correctly it's released now and it entirely supplants Ember. So how are you planning to gracefully sunset the project?

    Terence: I think locks (Ricardo Mendes) talked about how people already have five years of Ember experience, they can now move on to this Glimmer thing, right?

    Tom: That's right, yeah. You can put five years of Glimmer experience on your resume, on your LinkedIn profile. You know, something we really wanted to be mindful of is that it's really easy to think that we're giving up on Ember or that we just, declared bankruptcy and we’re starting over again fresh. Because actually, this is what happens in the JavaScript community all the time, right? New version, backwards incompatible, we decided that it was just too clunky to ever fix.

    Terence: Right. Angular 1 and Angular 2 thing?

    Tom: Something like that, right?

    Jonan: And in some cases, that's the right choice.

    Terence: Yeah, I think it is.

    Jonan: In the first version, there were mistakes made, let's move on. There is no right choice in those circumstances. Maybe it's the only choice that you have.

    Tom: Right. So Glimmer is a little bit different. The first thing to understand is that Glimmer is an extraction, it's not a brand-new library.

    One piece of feedback that we get all the time is people say, "You know, I would, theoretically, be interested in using Ember but I don’t need all of that stuff. I don’t need a router, I don’t need a data layer. I just want components. I have a Rails app and I just wanna do some kind of interactive widget." People use JQuery, they use React, things that are really easy to drop in, super simple to learn.

    So, we thought about it and said, "Well, you know, we actually have this awesome rendering library in this engine called Glimmer. Why don’t we make it available to people who don’t wanna buy into Ember?"

    You know it shouldn’t be an all-or-nothing thing. We should try to think about how we can bring incremental value to people. So that's one. It's not a new project. It's an extraction.

    The other thing is that I don’t think about Glimmer as being a different project. I think about Glimmer as being a way for us to experiment with the future of the component API in Ember. So one thing that we're working on right now, and actually there is an RFC written by Godfrey Chan, is an API that lets people write plug-ins that implement different component APIs.

    Remember, LinkedIn is an Ember app. It’s an Ember app that has had a lot of money and a lot of work put into it, and I promise you, we're not just gonna throw that away and rewrite it in Glimmer.

    So we really need to focus on the experience of taking Glimmer components and bringing them into an Ember app; that's what we're working on right now. Glimmer components, I think of it as the future of the Ember component API.

    What I would really love is that people can start working on Glimmer applications, see that it has this beautiful UI, it's fast, it's slick, all these things. Then they realize, "Hey, actually, maybe I need Ember data, maybe I need a router?" And then what they'll do is just take these Glimmer components, drag and drop them into their Ember app and, boom, they just work without having to change a line of code.

    Jonan: Ember includes a lot of things. It's prepared to handle problems that you can't foresee yet which is one of the benefits of using a framework. But that means that it's larger than maybe you need in the moment. So I could start with a very small Glimmer app, and Glimmer itself is small, right?

    Tom: Yeah, it's really small.

    Jonan: So the advantage right now though, because we don’t have that component transferability, we can't just take an Ember component and take it into Glimmer today, is that it’s small. You described it as useful for a mobile application. The example you gave in the keynote was a temperature widget that gave the temperature in various cities.

    Give us like a real-world use case of Glimmer.

    Tom: Sure. I mean I can give you a very real-world use case, which is that before I joined LinkedIn, I was working at this really awesome little startup in New York called Monograph. One of the things Monograph was doing was building these e-commerce apps that were designed to integrate into social media apps.

    So one thing that's really amazing about the web that you can't do on native, is you can actually run inside of other apps. You can run inside of Facebook, you can run inside of Twitter, you can run inside of any social media app and that's something that native apps can't do.

    What we wanted to do was build this experience that felt very native but it also had to load really, really fast. Because you didn’t get to load it until a user tapped the link. So we actually tried to build a few prototypes in Ember and they actually worked really great on the iPhone, but then we had some investors in Australia on Android phones, and when they tapped on it, it took like 10 seconds to load. That's just not acceptable when you're trying to target these mobile apps.

    I said "We have this great entry engine in Ember and it's really small. I wonder if I can hack together something using this?" And the reality was that if we couldn’t, I was gonna have to use something like React or something.

    So I told my boss "Give me a week. Give me a week to see if I can do it in Glimmer. I have this crazy idea, let's see if we can do it.", and we actually pulled it off.

    We've run a few campaigns now and things have sold out in like an hour. So the model works.

    I think if you're building an app that needs a router and uses a data layer, yeah, you should be absolutely using Ember. This is definitely a pared down experience, but my hope is that we're gonna figure out ways of taking these big apps and kind of slicing them up in a way that will be good for mobile.

    Jonan: I just want to make sure I’ve got the technical details right here. Ember rendered JavaScript in the past, and now it is rendering bytecode: a series of opcodes that are interpreted on the client side by a couple of different VMs. You have an update VM and you have a render VM. So the first time that you load up a page, you're gonna send over some bytecode and that's gonna be interpreted by this TypeScript VM, the render VM, and then the updates will come into the update VM in the next round? Okay. And so the actual content of this bytecode, what is that?

    Tom: It's a JSON object. The reason for that is, of course, JSON is a much smaller subset of the JavaScript language. So they're more compact and they're much faster to parse.

    Modern JavaScript VMs are very fast and very good at doing just-in-time compilation to native code. So if we emit JavaScript, those will get compiled into these really fast operations.

    The problem that we didn’t realize at the time was that when you have apps that grow, that is a lot of JavaScript, and all that JavaScript gets parsed eagerly. Now you are in the situation where you're spending all this time parsing JavaScript. For some parts of that page, it doesn’t need to get parsed because it never gets rendered.

    Jonan: So in the olden days, again, I need to simplify this for my own thinking here. I have a page with a home and an about page, right?

    Tom: Mm-hmm.

    Jonan: And I don’t ever click on the about tab. But that JavaScript is still there.

    Tom: It's still loaded.

    Jonan: And it's still interpreted.

    Tom: Still parsed, right.

    Jonan: And it's not necessary, right?

    Tom: Right.

    Jonan: So now, in this new world, the JSON blob that represents that about page, if the user never clicks on that link, it never actually has to get turned into anything.

    Tom: Right. We keep it in memory, resident in memory as a string, and we just-in-time JSON parse it. And of course, the JSON parsing is gonna be faster than the JavaScript parsing because of the fact that it's so restricted.

    Jonan: I see. And so then, you can take that JSON and turn directly into the page that you need. There's no other step there?

    Tom: Right.

    Jonan: I see, okay.

    Tom: So one of the advantages of Handlebars, you know, there's kind of this raging debate about if you should use templates like Ember, and Glimmer, and View, JSDO or should you use JSX like React and all of the stuff React likes.

    One of the really nice things about Handlebars is that it's very statically analyzable. We can very easily analyze your template and say, "Okay, these are the components you're using. These are the helpers that you're using. This is the structure." That lets us do more work when you deploy, when you build and that means less work on each user's browser.

    Jonan: Right, but then also, as you talked about in your keynote, maybe it comes at the expense of file size in some cases which is another problem that Glimmer solves. Because what you're sending over the wire is actually much smaller now.

    Tom: Right. So that was the other downside of doing the compilation with the JavaScript and that’s just the representation. I mean think about JavaScript as a syntax you have, you have functions, and you have var, and you have all of these different things. By creating our JSON structure, we can get that down.

    Jonan: So, we've been talking about progressive web apps a lot this year, and there are a lot of things that have happened recently that enabled progressive web apps to actually be a thing. We can now say reliably that you can present an offline experience that is close to the real experience with the possible exception of mobile Safari. I've heard that that's a popular browser.

    Tom: It is.

    Jonan: Something like 55% of the U.S. market is using an iPhone.

    Tom: That's right.

    Jonan: So they don’t have service workers that's the problem here, right? I wanna just explain real quick. A service worker, for my own thinking, is this thread that I can run in the background, and I can schedule tasks on it. So it doesn’t even mean you have to have my page open, right?

    Tom: Right.

    Jonan: I can go and refresh the data that I'm caching locally.

    Tom: The most important thing about the service worker, from my perspective, the thing that it unlocked in terms of taking something that usually only the browser can do, is now giving me, as a JavaScript programmer, access to intercepting network requests.

    Not just JavaScript but literally, I can have a service worker and if I put an image tag on my page and my service worker is consulted saying, "Hey, we're about to go fetch this image. Would you like to give me a version from cache?"

    That is hugely powerful when you're talking about building an offline experience. Because now you have programmatic access to the browser cache, to the way it looks at resources. So now, you have this very powerful abstraction for building whatever caching you want offline.

    Jonan: So whatever possible request could be coming from your application is more or less proxied through this server?

    Tom: Exactly. So in addition to their request, you also have access to browser cache. So you can put things in, you can take things out. That's what lets you program very specific rules. Because you don’t always wanna say use from the cache, right? Sometimes, there are things that you actually want like how many items in inventory remain, right? You probably don’t want that cached. You probably wanna have the most updated information possible.

    Jonan: We don’t have service workers in Safari and we won't for the foreseeable future.

    Tom: Well, we don’t have it in Safari but we have it in Firefox and we have it in Chrome. You know, the P in PWA, it stands for Progressive Web App, so you can progressively add this to your application. You know I think the best way to get features into a browser is to adopt them and say "Hey, if you're using an Android phone, you have this really awesome experience. But if you have an iPhone, you know, maybe it's not as awesome."

    Apple, I truly believe, really cares about the user experience. If there's one thing I've got from the Safari team is that they always prioritize making a feature fast and not draining the user's battery over being able to check a check mark.

    So I actually have a lot of respect for their position, and I think if they do service workers, they're going to do it right. If they see that people are having a better user experience on an Android phone than an iPhone that is hugely motivating for them.

    Terence: Does the service worker impact the batteries on phones?

    Tom: It could, it could, yeah. I think what browser vendors are going to have to figure out is what is the right heuristic for making sure that we can run a service worker, but only in service of the user, pardon the pun.

    How do we make sure that people aren't using it maliciously? How do I make sure this website is not mining bitcoin on your phone and now your battery life is two hours, you know?

    Jonan: Sure, yeah.

    Tom: It's a really tricky problem.

    Jonan: Even if they're relatively innocuous. They don’t necessarily need to be malicious. If you've got a hundred of them and they're all just trying to fetch the same image online, this will dramatically impact your phone's performance.

    Tom: Yeah, absolutely. Or if you think about, you know, you install a native app and all of a sudden, you start getting push notifications, that's not great for your battery life either.

    Terence: I guess, you talked about progressive web apps in last year’s keynote, what has the uptake been since then? I know it was kind of a work in progress kind of thing, and we just saw two talks yesterday related to progressive web apps.

    Tom: Yup.

    Terence: So has the adoption been pretty strong within the community?

    Tom: Yeah, absolutely. I think people are really excited about it. I think there are so many aspects to progressive web apps, and I think the definition isn't clear exactly. It's one of these terms that people talk about. Sometimes, it becomes more of a buzzword than a very concrete thing. There are a lot of things that you can do on the path to a progressive web app.

    So service worker, as Jonan said, is the one thing that people think about the most, but there are also things like server-side rendering, to make sure that the first few bytes that you sent to the user are in service of getting content in front of them. Not just loading your dependency injection library.

    Jonan: Right.

    Tom: You really wanna get the content first. There's the ability to run offline, there's the ability to add to the home screen as a first-class icon, the ability to do push notifications.

    Jonan: Removing the browser chrome, making it feel like a native app experience.

    Tom: Yup, and actually, Android has done some really awesome work here to make a progressive web integrate into the operating system such that as a user you can't really tell. That’s the dream.

    Jonan: Yeah, of course.

    Tom: The community uptake has been phenomenal, and this is exactly one of those things where it's gonna require experimentation. This is a brand new thing. People don’t know the optimal way to use it yet and that experimentation is happening.

    There are a ton of Ember add-ons: there are service worker add-ons, there are add-ons for adding an app manifest so you get the icon. All sorts of cool stuff.

    I think what we should start thinking about is, "Okay, well what is the mature part of this that we can start baking into the default experience when we make a new Ember app, such that you get a PWA for free?", and I would guess that we are probably on the way there, sometime this year or early next year. Saying that "You just get this really awesome PWA out of the box when you make a new Ember app."

    Jonan: That will be fantastic. I would like that very much.

    Tom: Defaults are important. I think if you care about the web, especially the mobile web being fast, the highest impact thing you can do is find out what developers are doing today and make the default the right thing.

    Terence: So do you imagine in the next couple years, PWA and FastBoot are just going to be baked into new Ember apps?

    Tom: I certainly hope so. I don’t think we want to do it before it's ready. FastBoot, in particular of course, introduces a server-side dependency.

    One thing that people really like about client side apps is that I don’t need to run my own server, I can just upload to some kind of CDN. That's nice, I don’t like doing ops. That's why I use Heroku so I don’t have to think about ops. So that's the hard thing about server-side rendering, it does introduce computation requirements when you deploy.

    So I don’t know if FastBoot will ever be the default per se, but I do know that I want to make it really easy and at least give people the option.

    "Hey, server-side rendering is really important for many kinds of apps. Do you wanna turn it on?" The PWA stuff, I think we can do it within the existing parameters of being able to do static deploys, so yeah, let's do it.

    Terence: If you have FastBoot on the app it’s totally optional though right?

    Tom: Yes, totally optional.

    Terence: You can still deploy the assets and ignore FastBoot completely, even if it was part of the standard app, right?

    Tom: That's true. Yeah, that's true, and really that, I think, is the beauty of client-side architecture plus server-side rendering. "Oh, my server is over capacity." Well, you can just have your load balancer fall back to the static site, and maybe the user doesn’t get the first view as fast but they still get the full experience.

    So much of what FastBoot is, is this conventional way of having not just the server-side rendering but also having a good user experience around it. So much of that relies on the good bits of Ember, the very conventional structure. So I think Glimmer will rapidly support server-side rendering but massaging that into an easy-to-use thing is, I think, an Ember responsibility.

    Jonan: The VMs that we're talking about, with the Glimmer on the frontend, the updated render VMs, are written in TypeScript.

    Tom: That's right.

    Jonan: You mentioned during your keynote that there were some features you added to TypeScript 2.2, or worked with the TypeScript team to add to TypeScript 2.2. and 2.3, to enable Glimmer? Is that me misunderstanding something?

    Tom: It's not enabling Glimmer per se, because Glimmer 2 from the beginning has been written in TypeScript. I think when they started TypeScript was on 1.8, so when you make a new Glimmer app, the default is to get TypeScript. That just works out of the box, because the library is written in TypeScript you get awesome code completion, you get intellisense, you get documentation in line, all these things automatically.

    I can't say enough positive things about the TypeScript team. They are so professional, they are so responsive. We even asked Daniel Rosenwasser, who is the PM, last week "Hey do you wanna come to EmberConf next week?" "I will come, because I really want to meet the Ember community." They're really, really wonderful.

    So for Glimmer, the internals, because it's written in Typescript, there were really no problems. But the thing that they realized is, "Hey, there's actually this long tail of libraries that come from earlier versions of JavaScript like when ES3 and ES5 were kind of cutting edge, that built their own object model on top of JavaScript."

    So if you look at Ember for, example, you have the Ember objects model where you have .get and .set, and you have ember.object.extend and ember.object.create. Before we had ES6 classes, we had no choice but to build our own on top of the language. The problem is we need some way to let TypeScript know, "Hey, when we call ember.object.extend, that's not some random method, that's actually defining a type. That's defining a class."

    The TypeScript team has been really awesome saying, "Okay, how do we rationalize that and add the extension points where a system like Ember or…" I mean here's the thing. Every JavaScript library from that era has their own system like this, so they've built these really awesome primitives in TypeScript that let you express key types or mapped types.

    "Hey, when you see ember.object.extend, we're gonna pass it to POJO, Plain Old JavaScript Object as an argument. That's not just a bag of data. I want you to actually look at the keys inside of that object and treat those like types."

    So that's the thing we're really excited about because, of course, you're going to be writing Glimmer apps, you're going to be writing Glimmer components.

    You're going to get these really nice TypeScript features but then we don’t want you to have to go back to Ember code and miss those benefits.

    Jonan: That's a fantastic feature to have in a language and it's a difficult thing to bring yourself to add, I would imagine, if you're maintaining something like TypeScript. I think this is a smart way to approach the problem.

    Tom: Yes.

    Jonan: But you're looking at all of these people with their own legacy object models and I have an object model now, and I want people to use the object model that exists in this language. Right?

    Tom: Exactly, yes.

    Jonan: How do I let you also just roll your own object model? It's a pretty fundamental part of a programming language.

    Tom: It is, yeah, and that' what I mean about professionalism. I really, really appreciate the TypeScript team thinking so carefully about adoption, because I think it really requires maturity to do that. How do we bridge the gap, reach people where they are today? And then we can slowly bring them into the new, modern world as they do new things. I think that's hugely important and I think it's one thing that many people in the JavaScript community undervalue. It is such a breath of fresh air to see it from Typescript.

    Jonan: That's great.

    Terence: Yeah. It seems to align a lot with all the stuff Ember tries to do with the way it does it features.

    Jonan: So at the very end of the keynote… You ran a little long on the keynote which is a rare thing to see.

    Tom: Yeah, yeah, very rare.

    Jonan: This year, you were overtime a little bit and you flipped through some content very quickly at the end. I was hoping maybe you could give us a peek at some of those things you didn’t get time to talk about in your keynote, that you wish you had time to mention.

    Tom: I think if we had had more time, one thing I would have really loved to go into more was the Glimmer API. I see the Glimmer API for components being the future of how you do components in Ember, and we have focused really hard on making these things feel like it's just regular JavaScript.

    Like I was saying, when Ember came of age, we didn’t have the ES6 classes. We couldn't even use ES5 because it wasn't adopted enough. So we built our own object model on top.

    Then rapidly, all of a sudden, the pace of JavaScript's development picked up, and now we have classes, and we have decorators, and we have getters, and we have all these amazing new things. Because it happened right after we stabilized our API, people look at Ember sometimes and think, you know, that feels like they're doing their own weird thing and already know JavaScript. It's like "I don’t wanna do it the Ember way. I wanna do it the JavaScript way."

    So what we tried really, really hard to do with Glimmer is say, "Okay, let's think about what someone who only knows JavaScript or modern JavaScript, what do they know and what are they expecting?" And let's just make the whole thing feel easy and natural for them.

    So for example, Glimmer component when you define it is just an ES6 class that extends the Glimmer component base class. The way that you import the Glimmer component is a standard import. Then there's a proposal in JavaScript called "decorators," which I believe is stage two. That lets you add certain annotations to properties, and methods, and classes and so on.

    Now in Glimmer we have introduced something called "track properties", but more importantly in Glimmer, you don’t actually need any kind of annotation because your computed properties are just getters, which is built in the language. Of course, if you want to do change tracking like "Hey, this computed property changed, how do I update the DOM?" You have a very simple decorator. So you don’t have to have this weird Ember thing, you just do what's in the language.

    Jonan: Which is hopefully going to increase adoption.

    Tom: I hope so, yeah.

    Jonan: This is a common problem, not just in the JavaScript community. You're coming up with new frameworks and you're moving very quickly. JavaScript, in particular, is moving very quickly. It seems like every week, or month, there's some new tool that I would have to learn, right?

    Tom: Yeah.

    Jonan: Something new and each one of them has their own distinct syntax, constantly changing. If you keep moving the goal post, eventually people tire of it. I consider the approach you took with Glimmer to be a very mature approach, and I really appreciate the effort you put in to make that.

    Tom: I think when people see Glimmer, it's very easy for their reaction to be "Oh, god, here comes another JavaScript library." What I hope is that people can look at our track record, and I hope we have some credibility with people, and see that, "Hey, we're not just talking a big game here. We actually have a community that has gone back at least five years. And we have apps that are five years old that have migrated."

    So I just hope people can feel safe when they look at Glimmer. It checks all the checklists that you need to check in 2017, but it also comes with the same community and the same core team that really values stability, that values migration, that values convention.

    Jonan: And speed.

    Tom: Yeah, and speed.

    Jonan: I think speed is the real reward from Glimmer. You build something in Glimmer and you, somehow, have accomplished this impossible tradeoff where you have a fast render speed and a fast update speed.

    Tom: I think it's interesting too because, you know, this always happens with benchmarks. There's some suite of benchmarks that comes out, people become over-focused on one particular metric.

    Jonan: Right.

    Tom: In this case, the community, has really focused on, in the last year, initial render performance. Initial render performance is super, super important, but it's not always worth sacrificing updating performance. I think Glimmer has hit this really nice sweet spot where it’s not as absolutely fast as the fastest rendering library, in terms of initial rendering, but it blows away all the other rendering engines at updates.

    Being the absolute fastest at initial render is only so important, so long as the user notices. It's not worth sacrificing everything if your constant time is imperceptible to the human, and I'm really excited with that sweet spot that we've hit.

    Jonan: We were talking the other day at lunch about the fact that there are some pages where I really don’t mind a long load time. If I'm going to a dashboard for a product that I've already purchased, I'm gonna sit there and wait. Like, yeah, maybe it takes 10 seconds, right, and I'm gonna be super annoyed and think, "Wow, why am I paying these people money?" Right? But for some definition of fast, all things start to be equal, when we get down towards those lower numbers.

    Tom: That’s right and I think people conflate those. You know, it's easy to get in a Twitter flame war because I'm talking about my dashboard that people are gonna sit on all day. You're talking about this ecommerce site. If you don’t have a response in under 200 milliseconds, people are gonna bounce and you're not gonna make your money. So those are different categories.

    That being said, I really do believe in my heart that there is a future where you can build your big dashboard app and it doesn’t take forever to load if we make the tools really good.

    Jonan: Thank you so much for taking the time to talk to us today. I really appreciate it. Do you have anything else you wanna share? Last minute thoughts?

    Tom: Oh, I just cannot wait to take a vacation in Barbados for a week.

    Jonan: Tom, thank you so much for being here.

    Tom: Thank you, Jonan, and thank you, Terence.

    Terence: Thank you.

  • The History of Ember.js: An Interview With Tom Dale at EmberConf - Part One (Heroku)
    09 May 2017 14:51

    At EmberConf Terence Lee and I had a chance to sit down with Tom Dale and chat about the history of Ember.js and where it’s headed now, including some details on the newly extracted Glimmer.js rendering engine. This post details a lot of the history of Ember, including some of the motivation that led the framework to what it is today. Watch the blog for the second portion of this interview with all of the details on Glimmer.js. The next post will also include the full audio of the interview, with many questions we opted to omit from the transcription to save valuable bytes.

    Jonan: So, we're at EmberConf speaking with Tom Dale, who gave a keynote today with some important announcements. We're going to dig into those in just a minute here, but I’d like you to introduce yourselves please.

    Tom: Sure. Hey, I'm Tom. I just started working at LinkedIn as a senior staff software engineer, and I work on a really awesome team that works on Ember infrastructure. As you may have seen, LinkedIn’s website now is one big Ember application. So my job is to make the army of engineers at LinkedIn productive, and make sure that we're able to build a really awesome web software.

    Terence: I'm Terence, I do language stuff and Rails on the languages team [at Heroku].

    Jonan: There's a third-party Ember buildpack that you worked on, right?

    Terence: Yes. That has no JavaScript in it.

    Jonan: No JavaScript at all? But it ships Ember. I shipped my first Ember app on it.

    Tom: That's not true.

    Terence: It is true.

    Tom: It's all Ruby?

    Terence: Oh, yeah.

    Tom: Awesome. See that's great. You know what, Ember is a big tent, as DHH would say. Not about Ember, he would say that about Rails and then I would copy that because that's basically what we do. We just take what DHH says, and we repeat them in the context of JavaScript, and it sounds very thought leadery.

    Jonan: Would you describe Ember as Omakase?

    Tom: I would describe it as being bespoke, artisanal, shade-grown Omakase.

    Jonan: That's even better. So on the subject of Ember.It's been around for awhile now. How old is Ember? Five years plus?

    Tom: It depends on what date you want to use. So if you're talking about Ember 1.0, I think it's been about five years.

    Terence: Do you include SproutCore in that?

    Tom: I mean I think we should. There is no Ember without SproutCore, and to me SproutCore was one of the first libraries or frameworks to adopt this idea of client-side architecture. So one thing that we talked about in the keynote yesterday was just how much the web has changed in five years, right? So five years ago, IE was the dominant browser but actually, SproutCore had it way worse. And we're talking about IE6 and IE7 and talking about ambitious things, what we do on the web.

    Jonan: And you did it in an era where browsers were not even close to where they are today.

    Tom: Not even close, not even close.

    Jonan: That's interesting. So then, from SproutCore, Ember comes out five years ago and we're off to the races. A lot changed in that first year, you went 1.0 and you’ve said that there were a lot of things that went wrong along the way. In your talk, you had a slide where you mentioned a few of those things. From the 10,000-foot view, what kind of lessons did you learn in those first 5 years?

    Tom: JavaScript apps felt broken and people didn’t know why but people always said, "JavaScript apps feel broken, you know, for whatever reason, please don’t use them" right? And people wanted to shame you for using JavaScript. The reason for that, I think, is URLs. URLs are kind of the linchpin that holds the web together. And so much of the value of the web over native is these URLs, and JavaScript apps just ignored them. SproutCore ignored them, and almost every JavaScript framework did. So, what Ember had to do was figure out how to build JavaScript apps that don’t feel broken on the web. That’s where all this work with the router started.

    Nowadays, routers are taken for granted. Every framework, every library has a router that you can drop into it. But there was actually some novel computer science work that went into it, in how we tie the architecture of the app to the URL. That took a long time and it was a very organic process. I don’t think we fully understood the magnitude of the research project that was going on. There are a lot of examples of that where we tackled problems for the first time, so of course, there's gonna be kind of an organic exploration of that space.

    Another example of this is that when we adopted the six-week release cycle, this train model with Canary Beta and the release, the only other people doing it were Chrome and, I think, Firefox. And when we adopted it, it paid dividends right away, and I'm so happy that we adopted it. One constraint that we have that Chrome and Firefox don’t have as much is that for us, we're always shipping the framework over the wire every time a user visits your webpage, right?

    Jonan: Right.

    Tom: So it's very easy to have feature flags and to keep all the APIs around when you're distributing a binary. It's much harder when every time you do that, your file size goes up, and up, and up. And so what we've had to figure out is okay, "Well, we really liked this release train model. People really like the fact that it's backwards compatible. People really don’t like ecosystem breaking changes like Python 2 to Python 3 or Angular 1 to Angular 2. That doesn’t work so what do we do?"

    You know, you feel kind of stuck. So we've had to figure out a lot of things. Like one thing that we've been working on is something called Project Svelte, which is the ability to say, "You can opt out of deprecated features and we will strip those from the build".

    Jonan: But that's the only way that you can really move forward there. I mean if you've got to make this smaller, you can't just deprecate things arbitrarily.

    Tom: Right.

    Jonan: You can't make those decisions for your user. Your file size is ever growing, which when you're shipping over the wire, is not a great thing.

    This has already, historically, been an issue for Ember, the size of the framework.

    So what you are providing people now is a way to opt out of some those deprecated features. So say that, "All right, I've stopped using this API in my codebase, we can strip this out."

    That's known as Project Svelte?

    Tom: Yeah, that's Project Svelte. It's really important to remember that when Ember started, there were no package managers. NPM wasn’t 1.0 or just hit 1.0, and was not at all designed for frontend packages. It didn’t do any kind of deduplication and distributing.

    This is back in the day when the way that you got a library was you Googled for the website, you found it, they gave you a script tag to just drop in. I'm sure you all agree that's a horrible way to do dependency management.

    So we felt compelled to say, "Well, if we wanna make something… If we want people to actually use something, we have to bake it in." Because when you're gathering all your dependencies by hand, you're only gonna have, you know, four or five of them. You're not gonna go get a million dependencies. Of course, that has changed dramatically and we have new technology like Yarn, which is more like a Cargo/Bundler style of dependency resolution for JavaScript.

    What we found has not worked is trying to do big-design, upfront projects, because anything that we land in Ember gets that guarantee of stability and compatibility.

    People feel a very strong sense of responsibility, that if we land this feature, this has to be something that we are ready to support for the foreseeable future, and that just takes longer. It's the same reason standards bodies move relatively slowly.

    Jonan: Right. Now, this is something you brought up in your keynote. Rather than architecting or spending a huge amount of time and investment upfront architecting your system, you want to get it out in front of the customers as early as possible. But that conflicts with the idea that you're trying to present stable products, things that won't change, right?

    Terence: Stability without stagnation is the tagline right?

    Tom: Right. So that's the message but then we also know that you can't do a big design upfront, and you're not gonna get it perfect the first time. You ship an MVP and iterate.

    So how do you balance this tension? If you look at the projects we've embarked on in the last couple of years, there have been some projects that were more big design upfront. And those have largely stagnated and failed because of the fact that we just couldn’t get consensus on them.

    Then you have some other projects like Ember Engines and FastBoot. What we actually did was look at how web standards bodies work -- CC39, W3C, WHATWG.

    There's something called the "Extensible Web Manifesto," which you may have seen, that says "Hey, standard bodies, open source libraries are able to iterate a lot faster than you are. So instead of focusing on building these big, beautiful, really-easy-to-use declarative APIs, give us the smallest primitive needed to experiment on top of that."

    That’s something that we really take to heart in Ember 2. If you think of Ember as being this small stable core, what we can do is expose just the smallest primitive that you need, and then we can let the experimentation happen in the community.

    So with FastBoot, for example, FastBoot is this entire suite of tools for deploying server-side rendered, client-side apps. You can easily push it to Heroku and, boom, it starts running, but that doesn’t need to live in Ember. We can do all the HTTP stuff, all of the concurrency stuff. All of that can live outside of Ember, all Ember needs to say is, "Give me a URL and I will give you the HTML for that back."

    So that's what we did. There's this API called Visit, the ‘visit’ method. You call it, you give the URL, you get HTML back, and it's so simple and you can easily have discussion about it.

    You can understand how it's gonna operate and that's the thing that we landed. Then that's given us a year to experiment in FastBoot and make a lot of really important changes.

    Jonan: You were able to hide the complexity away behind this simple API.

    Tom: Right.

    Jonan: So some of the things that more recently you mentioned in your keynote as not having gone well, were Ember Pods, for example, and now we have Module Unification. So if I understand correctly, Ember Pods was a way to keep all of your component files related to a single component in one location?

    Tom: Right. The Rails style where you have one directory that's all controllers and one directory that's all views or templates, which is how Ember started. It's still the standard way, the default way you get when you create a new Ember app.

    People found it more productive to say, "I'm gonna have a feature directory", where you have your component and that component might have style. It might have JavaScript, it might have templates. I think it's just easier for people to reason about those when they're all grouped together, instead of bouncing around.

    Jonan: I love this idea. When I first came into Rails, I distinctly remember going from file to file and thinking, "Where even is this thing. How do I find this?"

    So you had said that Ember Pods, maybe, didn’t seem to take off? It wasn't a very popular solution to that problem, and now we have Module Unifications. How is that different?

    Tom: I actually think that Pods was popular, it actually was very popular. So, there's something DHH says: "Beginners and pro users should climb the mountain together."

    I think it's a bad sign, in your framework, if there's the documented happy path that beginners use, and then at some point, they fall off the cliff and see "Oh, actually there's this pro API. It's a little bit harder to use but now that you're in the club, now you get to use it". I think that leads to very bad experiences for both. You kind of wanna have both sets of people going up the same route.

    So Pods is almost this secret opt-in handshake. And it was just one of those things where it started off as an experiment but then slowly became adopted to the point where, I think, we didn’t move fast enough.

    Jonan: I see.

    Tom: We didn’t move fast enough and now, there's almost this bifurcation between apps that are not using Pods and apps that are using Pods.

    So with Module Unification what we did is we sat down and we said "OK, Pods was a really nice improvement but it didn’t have a ton of design applied to it. It was the kind of thing that evolved organically. So let's just sit down and try to design something."

    For us, it was really important with Module Unification to say, "Not only does it need to be good but we need to have a way of being able to automatically migrate 99% of the Ember apps today. We should have a command that will just migrate them to the new file system."

    So one thing that's really neat is that you can just have a component where all you have to do is drag it into another component's directory and now it's scoped. It's almost like a lexical scope in a programming language. We're using the file system to scope which components know about each other.

    Jonan: So, forgive my simplification here but I'm not great at Ember. If I have a login component and it's a box to just log in, and inside of it I wanted to have a Google auth button and a Twitter auth button, each of those could be independent components.

    Maybe I wanna reuse it somehow. I would drag them into my login directory and that makes them scoped, so we can't use them somewhere else.

    Tom: Right. That ends up being pretty nice because often, what you'll do is you'll create a new component, give it a really nice and appropriate semantic name and, oops, it turns out your coworker used that for another page, a year ago. Now, you can't use it, because it’s completely different.

    Jonan: So I've got my Ember app and I've been using Pods all this time, and now, we have Module Unification and there's a new way to do this. I can just move over to module unification right?

    Tom: Yes.

    Jonan: We run this script that you've written and it would migrate me over?

    Tom: Yeah. So we have a migrator and because there's so many Ember apps using the classic system, so many Ember apps using the Pod system, it can handle both.

    Terence: Could Module Unification have happened without Ember Pods happening first?

    Tom: It's hard to say. I think it's something that people really wanted, and I think it's fantastic. This is something we touched on the keynote; one thing that we've always said about Ember, and I think this is true about Rails also, is that there's always a period of experimentation when something new comes along. You really want that experimentation to happen in the community. Then eventually, it seems like one idea has won out in a lot of ways. The things that we learned about with Pods fed directly into Module Unification design.

    Jonan: So maybe, we could chat a little bit about deprecating controllers in Ember?

    Tom: Sure, yeah.

    Jonan: You announced that you were going to deprecate all of the top-level controllers by 2.0, and then pushed 2.1 and 2.2. That's still the plan to deprecate the controllers someday?

    Tom: I think what we are always dedicated to is trying to slim down the programming model and always reevaluate what is the experience like for new people. I don’t want to say that we're going to deprecate controllers because that sounds like a very scary thing, right? There's a lot of people with a lot of controllers in their apps. But I do think what we will want to do is take a look at the Ember programming model from the perspective of a new user. And say, "Well, it seems like people already learned about components. And it seems like there's probably some overlap between what a controller does and what a component does."

    So maybe there's some way we can unify these concepts so people don’t have to learn about this controller thing with its own set of personality quirks.

    Jonan: Is this where routable components fit into the idea then?

    Tom: So that's the idea of routable components and I think I don’t have a concrete plan for exactly how this is going to work. I think a lot of ways, the work that we want to do on that was blocked by the Glimmer component API.

    I think what we'd like to do is add whatever low-level hooks in Ember are needed so that we can maybe do some experimentation around things like routable components outside. Let people get a feel for it and then once we have a design that we're really happy with, then we can land it back in mainland Ember.

    That’s the end of our discussion on the history and direction of the Ember project. Stay tuned for part two and learn more about the Glimmer.js project.

  • Hello RedBeat: A Celery Beat Scheduler (Heroku)
    02 May 2017 15:32

    The Heroku Connect team ran into problems with existing task scheduling libraries. Because of that, we wrote RedBeat, a Celery Beat scheduler that stores scheduled tasks and runtime metadata in Redis. We’ve also open sourced it so others can use it. Here is the story of why and how we created RedBeat.


    Heroku Connect, makes heavy use of Celery to synchronize data between Salesforce and Heroku Postgres. Over time, our usage has grown, and we came to rely more and more heavily on the Beat scheduler to trigger frequent periodic tasks. For a while, everything was running smoothly, but as we grew cracks started to appear. Beat, the default Celery scheduler, began to behave erratically, with intermittent pauses (yellow in the chart below) and occasionally hanging (red in the chart below). Hangs would require manual intervention, which led to an increased pager burden.


    Out of the box, Beat uses a file-based persistent scheduler, which can be problematic in a cloud environment where you can’t guarantee Beat will restart with access to the same filesystem. Of course, there are ways to solve this, but it requires introducing more moving parts to manage a distributed filesystem. An immediate solution is to use your existing SQL database to store the schedule and django-celery, which we were using, allows you to do this easily.

    After digging into the code, we discovered the hangs were due to blocked transactions in the database and the long pauses were caused by periodic saving and reloading of the schedule. We could mitigate this issue by increasing the time between saves, but this also increases the likelihood that we'd lose data. In the end, it was evident that django-celery was a poor fit for this pattern of frequent schedule updates.

    We were already using Redis as our Celery broker, so we decided to investigate moving the schedule into Redis as well. There is an existing celerybeatredis package, but it suffers from the same design issues as django-celery, requiring a pause and full reload to pick up changes.

    So we decided to create a new package, RedBeat, which takes advantage of the inherent strengths of Redis. We’ve been running it in production for over a year and have not seen any recurrences of the problems we were having with the django-celery based scheduler.

    The RedBeat Difference

    How is RedBeat different? The biggest change is that the active schedule is stored in Redis rather than within process space of the Celery Beat daemon.

    No longer does creating or modifying a task require Beat to pause and reload, we just update a key in Redis and Beat will pick up the change on the next tick. A nice side-effect of this is it’s trivial to make updates to the schedule from other languages. As with django-celery, we no longer need to worry about sharing a file across multiple machines to preserve metadata about when tasks were last run. Startup and shutdown times improved since we don't suffer from load spikes caused by having to save and reload the entire schedule from the database. Rather, we have a steady, predictable load on Redis.

    Finally, we added a simple lock that prevents multiple Beat daemons from running concurrently. This can sometimes be a problem for Heroku customers when they scale up from a single worker or during development.

    After converting to RedBeat, we’ve had no scheduler related incidents.


    Needless to say, so far we’ve been happy with RedBeat and hope others will find it useful too.

    Why not take it for a spin and let us know what you think?

  • The Heroku-16 Stack is Now Generally Available (Heroku)
    20 Apr 2017 15:06

    Your Heroku applications run on top of a curated stack, containing the operating system and other components needed at runtime. We maintain the stack - updating the OS, the libraries, and ensuring that known security issues are resolved, so that you can focus on writing code.

    Today we're announcing the general availability of Heroku-16, our curated stack based on Ubuntu 16.04 LTS. In addition to a new base operating system, Heroku-16 is updated with the latest libraries. If you’re a Ruby or Python developer, Heroku-16 includes 15% more development headers at build time, making it easier to compile native packages on Heroku. Finally, Heroku-16 offers a better local development experience when using Docker, because of its smaller image size.

    Since its beta in March, Heroku-16 has been tested on thousands of applications and is now ready for production on both Common Runtime and Private Spaces apps. Heroku-16 will become the stack new applications use (i.e., the default stack) on May 8th, 2017. To learn more about testing and upgrading your app, check out the Heroku-16 documentation.

    What's New

    Smaller Docker Image

    With the release of Heroku-16, we’ve changed the architecture of the stack, allowing us to provide you with a curated Ubuntu 16-based Docker image at 465 MB (vs 1.35 GB for Cedar-14).

    To use Heroku-16, specify it as your base image in your Dockerfile:

    FROM heroku/heroku:16

    By using the Heroku-16 Docker image for local development, you ensure the stack running locally is the same stack running on Heroku (i.e., dev/prod parity). Everyone -- Heroku customer or not -- is free to use the Heroku-16 Docker image.

    Improved Support for Compiling Native Ruby and Python Packages

    At build time Heroku-16 includes 15% more development headers than Cedar-14. This means fewer failed builds when your app needs to compile native Ruby or Python packages.

    Updated Stack Libraries

    Heroku-16 should largely be backwards compatible with Cedar-14. We have, however, removed lesser used packages to reduce the security surface area and stack image size. Apps may also encounter incompatibilities because libraries on Heroku-16 have been updated to their most recent versions. Learn more about the packages installed in Cedar-14 and Heroku-16.

    How to Test and Upgrade

    Testing Heroku-16 with your application, especially if you use review apps, is easy. Simply define your stack in app.json and create a new pull request:

       "stack": "heroku-16"

    If your tests are successful, you can upgrade your application:

    $ heroku stack:set heroku-16 -a example-app
    $ git commit -m "upgrade to heroku-16" --allow-empty
    $ git push heroku master

    For more information on upgrading your app, check out the Heroku-16 documentation.

    Stack Support

    Heroku-16 is now generally available and we recommend you use it for new apps. Heroku-16 will be supported through April 2021, when Long Term Support (LTS) of Ubuntu 16.04 ends. Cedar-14, the previous version of our stack, will continue to be supported through April 2019. For more information, check out our stack update policy.

  • On Building Tools for Developers: Heroku CI (Heroku)
    18 Apr 2017 16:24

    How we built Heroku CI: our product intuition checked against what the market wants (we surveyed ~1000 developers to figure out the latter, and the results were surprising)

    Two approaches to building any product are often in tension: designing from inspiration, and designing from information. On the pure inspiration side, you just build the product you dream of, and trust that it will be so awesome and useful, that it will succeed in the market. On the pure information side, you build exactly what the market is asking for, as best you can tell (think: surveys, top customer feature requests, consultants, customer panels).

    Our initial design for Heroku CI (currently in public beta) was nearly pure inspiration, in large part from our engineering staff. We had a good idea of our dream CI product, and many of the raw materials to build it (Heroku apps, our build process, Heroku add-ons, etc.). And Heroku, like the rest of Salesforce, strongly encourages such experimentation and innovation when individual product teams are so inspired. We create and improve a lot of amazing products that way.

    Heroku CI: Design by Inspiration

    The Heroku DevEx (Developer Experience) team is distributed across two continents and four countries. We need well-structured, team-oriented developer workflow as much as our users and customers, including, of course, running tests for continuous integration (CI). And we are a pretty experienced team of developers, product, and design. We spend a lot of time with customers, and, of course, live and breathe in today's app dev and DevOps world.

    So the Heroku CI design began with our own well-formed and granular ideas on what our customers wanted out of CI, and we quickly arrived at what turned out to be a pretty stable initial design.

    Our alpha launch had Heroku CI integrating with, and extending, Heroku's continuous delivery (CD) feature set: Heroku Pipelines, Review Apps, and Heroku GitHub deploys. The initial feature set and developer experience were inspired largely from our own intuitions, experiences, and conceptions of customer need.

    Heroku CI: CI Design by Information

    Once we had a limited-functionality alpha version (MVP of our pre-conceived design for Heroku CI), we tested it over several weeks internally, then invited Heroku users via Twitter and direct e-mail to try it out.

    For users to access this "limited public beta" CI feature, they had to complete a survey on how they use CI and what they want from CI. We wanted to determine how they use (and hope to use) CI, and also wanted to make sure those interested could actually use the new Heroku CI alpha -- e.g. that we supported the language they required.

    As it turned out, so many users responded -- about 1000 for most questions -- that our short survey has become the largest of cloud PaaS developers on CI in recent years. The data helped us shape the planned CI feature set for public release. We also think the survey results will help the dev tools community understand how developers want to use CI, and so we are making the full results available in this post.


    The survey covers customers' responses on how they want to use CI, and how it fits into their CD and team workflows.

    You can read elsewhere about using Heroku CI. And you can just try it out yourself: the feature is in public beta and available by default to all Heroku users (just check the Heroku Pipelines web interface). Right now let's talk about how we decided what to build.

    Some customer concerns initially seemed obvious, but we asked anyway: of course we found that CI users want shorter queue times, faster test runs. Some features were not as predictable. And some of what's been most successful in our private beta was not requested by surveyed or otherwise vocal users at all.

    How Companies Build Successful Devops Products

    Almost a year ago, at the Heroku DevEx team offsite, we met up one morning at a London coffee house, the café at the Hoxton, Shoreditch. 77915323_shoreditch An informal conversation on what features/products to build next turned to CI. There had been customer requests to be sure. And it seemed to us like CI would be a great addition to our Heroku Flow continuous delivery features. Most of all, we just wanted to build it. CI seemed to us so compelling for our users as an integrated feature, and so compelling to us as a very cool thing to build, that there was soon consensus on the team. I'll note here that even at a biggish company like Heroku our team excitement around building cool stuff that users will love has a lot of influence over what we build. We didn't do a ton of financial analysis, or call our big-firm Analyst. We, the Developer Experience Team, wanted to build CI; there was roughly adequate customer and strategic reasons to do it, and we had the collective excitement and motivation to give it a shot. Personally, it’s gratifying to me that software can be built that way at Heroku and Salesforce.

    The Hoxton café, by the way, is a fabulous place. You can stay there all day, and we did, occupying valuable retail space and working out what Heroku CI might look like: UX, features, integrations, and above all, how the feature will live up to the Heroku promise of being (all at one now) simple, prescriptive, and powerful. Thankfully they serve breakfast, lunch, and a very nice dinner.

    On the Rise of DevOps



    CI as a practice in software development has been growing relentlessly over the past decade (even as interest in software development as a holistic discipline has stagnated). "Software development" is increasingly many separate disciplines, each with its own constituency and community -- the DevOps segment alone is projected to be a nearly $3 billion market in 2017. And developer focus on DevOps has given rise to many great products in the CI market. We wanted to understand how we could help developers have even better CI experiences, get better productivity, and build better products of their own.

    Our spike on the idea of a "Heroku CI" was essentially a simple test runner (for Ruby and Node) with a nice-looking UI integrated into the Heroku Pipelines interface. This spike, plus a nice UI and some basic supporting features, constituted our the version we released to alpha. Being Heroku we had a lot of the parts we needed to assemble test environments (ephemeral apps, dynos, add-ons, a build system, …) just lying around, and lots of co-workers willing to try out what we built.

    Quite quickly, we had an initial CI test-runner bolted onto the existing Pipelines feature. We found it useful to us internally and we felt comfortable inviting a few dozen friends-of-Heroku to use it and give feedback. Within a few more weeks, and a few dozen iterations, we were ready to ask a broader audience to weigh in (and take the survey).

    On the Rise of Pre-integrated Devtools Offerings?

    Yes, CI is integrated with Heroku Pipelines, and in fact they share a UX. While we could separate these in the future, the integrated offering is proving popular with users from big enterprise to small teams. We think this is in large part because it’s integrated.

    There was a time when "best of breed" was the catchphrase in DevOps. You spun together VMware, Chef, Puppet, Jenkins, Git, etc. So why are integrated offerings popular now? Our thinking is that CI/CD go together like chocolate and peanut butter: it’s a bit messy to put together yourself, but great when someone packages it for you. We noticed that our users, in general, enjoyed using CI products that set-up and integrated easily with Heroku Pipelines (great products like Travis and Circle). The popularity of fully integrated CI offerings wasn't lost on us either (look at the popularity of GitLab's integrated CI, or Atlassian's increasingly integrated devtools products). Among other advantages, you get single auth, add-ons work automatically, and there's no context switching from product to product in use or administration.

    The Consumerization of DevOps

    As importantly, we've noticed developers are demanding better UX for dev tools in general, and CI in particular (think of the delightful interface of Snap CI and others). We also noticed the pain many users described in setting up Jenkins, which leads the CI industry, but comes with a dizzying array of complex options, plug-ins (1500+), and a typically long set-up time, and maintenance labor. An extreme but real example here: one large Heroku customer needs to open a ticket with its IT department to provision each new Jenkins instance, often adding two months to the start-up of a new project if they want to use CI. This company is now using integrated Heroku CI on some projects, reducing CI setup time from months to minutes.

    The Customer Survey Results (and How We Applied Them)

    Preferred Developer Stack for Heroku Developers

    Developers, statistically, are centering around a few preferred workflow stacks. GitHub and Slack are the dominant VCS and team chat[ops] tools by most current market measures -- strong integration with these seemed necessary for us, and for any DevOps product. Atlassian's Bitbucket, HipChat are each a solid second in these roles. Together with Trello this would seem to give Atlassian a significant enough center of gravity (especially at larger companies) to also require good integration. And GitLab is surging in VCS and CI (and in developer buzz), at a curve that seems poised to challenge much longer-standing products. Being part of these popular toolchains is important to us, and where there is overlap in features, developers can choose where they want to live on the way to deployment and delivery.

    Note that, as Heroku is a cloud platform, our user base tends to prefer cloud products. We know also that a significant proportion of our customers who use "on premise" version control options like GitHub Enterprise and Bitbucket Server are hosting them in the cloud, often on AWS. So even a portion of these on-prem components are running as a self-managed cloud-like service.


    Simple Integrated Solutions Over Best-of-breed Complexity

    Jenkins is the leader in CI market share (around 70% by most analyst estimates). And its usage is growing at a pretty good clip. The vast majority of Jenkins installations are on-premise (about 83%) Our customers are always in the cloud, deploying to either Heroku's Common Runtime -- our traditional secure public cloud Heroku -- or to Heroku Private Spaces, our virtual private PaaS cloud offering.

    So for the cloud customers, as in our survey, the CI product usage sizes out a bit differently:


    I noted earlier the popularity of GitLab's integrated CI being significant to us in our decision to integrate CI. GitLab's been really good at moving fast in bringing popular, highly integrated features to market, building on its popular VCS. Note here that GitLab CI it is clearly the biggest mover in activity on Stack Overflow among popular cloud CI solutions.


    But all these cloud CI solutions are still dwarfed by Jenkins (as noted above, only 17% of Jenkins installs are in the cloud).


    Interestingly, Jenkins’ lovely new "Blue Ocean" UX, courtesy of CloudBees, seems to underscore the growing importance of simple, prescriptive developer experience in modern CI/CD solutions -- something that has always been, and is still, job 1 at Heroku.

    Speed Matters - A lot

    You'll notice in the first chart this post that fast queue time and fast execution time are the top most important features for surveyed users other than price.

    We have eliminated queue time altogether with Heroku CI. A new ephemeral CI environment (oh: it’s a Heroku app) is created at the beginning of each test run, and then destroyed when tests are complete. While there is some nominal setup time here, it’s nothing like traditional queue time waiting for a free CI worker.

    Environment Parity: Close Enough Is Good Enough for Most

    Note that at the chart at the top of the post, only about 40% of respondents found production environment parity to be "important". When we initially conceived of Heroku CI, we thought one of the biggest selling points would be environment parity between test runs environments (apps) and staging and production apps (environments). To our surprise, users did not care as much as we thought they should.

    This fact led us to an innovation around add-ons for ephemeral environments like CI and review apps. We now inform add-on partners that their add-on is being provisioned for an ephemeral Heroku app (a CI app in this case). During provisioning, the add-on vendors can choose to eliminate slow/costly features of the add-on that might take a long time to spin up, but are usually unnecessary for apps that will exist for such a short time – like long-term logging, billing features, etc.

    In this way, we are working across the ecosystem of cloud-based services that developers use to make sure each component is automatically configured for CI.


    We did make sure that you can customize CI apps in the environments section of the newly revised Heroku app.json manifest. Customizability, when we can do it in a way that doesn't complicate, is as important as being prescriptive.

    And Some Most Requested Features We Under-predicted


    Re-running tests was not part of initial CI alpha, and we were somewhat surprised by the high number of people who requested it both in the initial survey and among alpha users. So there’s now a big "Run Again" button on the UI, and we do find this feature frequently used in the current public beta.


    Browser testing – often called user acceptance testing or UAT – was popular with surveyed users, so it was moved up in our list of features planned for launch. UAT in a browser is also required by many Salesforce developers, whose applications are often web-deployed and/or exist within the Salesforce interface. (note that Heroku CI will also be used by Salesforce developers using Apex, hybrid-language apps, or frameworks like Lightning.)

    The Takeaway

    Product design by inspiration is exhilarating, and it's great to see a product or feature that arises from the sheer excitement of the team to build it succeeding with users. Verifying that our intuition was right with users takes a lot of time, but it's well worth it. In short what worked for us was to trust our instincts, but verify with users (and adjust).

    Most importantly, we view the developer community as a diverse ecosystem of innovators, thought leaders, vendors, and open source efforts, among others. It's a pleasure to share what we know, learn, and create with our vibrant, diverse community.

    Let us know what you think of Heroku CI, and what you want in your CI solutions, at

  • Sockets in a Bind (Heroku)
    30 Mar 2017 18:00

    Back on August 11, 2016, Heroku experienced increased routing latency in the EU region of the common runtime. While the official follow-up report describes what happened and what we've done to avoid this in the future, we found the root cause to be puzzling enough to require a deep dive into Linux networking.

    The following is a write-up by SRE member Lex Neva (what's SRE?) and routing engineer Fred Hebert (now Heroku alumni) of an interesting Linux networking "gotcha" they discovered while working on incident 930.

    The Incident

    Our monitoring systems paged us about a rise in latency levels across the board in the EU region of the Common Runtime. We quickly saw that the usual causes didn’t apply: CPU usage was normal, packet rates were entirely fine, memory usage was green as a summer field, request rates were low, and socket usage was well within the acceptable range. In fact, when we compared the EU nodes to their US counterparts, all metrics were at a nicer level than the US ones, except for latency. How to explain this?

    One of our engineers noticed that connections from the routing layer to dynos were getting the POSIX error code EADDRINUSE, which is odd.

    For a server socket created with listen(), EADDRINUSE indicates that the port specified is already in use. But we weren’t talking about a server socket; this was the routing layer acting as a client, connecting to dynos to forward an HTTP request to them. Why would we be seeing EADDRINUSE?

    TCP/IP Connections

    Before we get to the answer, we need a little bit of review about how TCP works.

    Let’s say we have a program that wants to connect to some remote host and port over TCP. It will tell the kernel to open the connection, and the kernel will choose a source port to connect from. That’s because every IP connection is uniquely specified by a set of 4 pieces of data:


    No two connections can share this same set of 4 items (called the “4-tuple”). This means that any given host (<SOURCE-IP>) can only connect to any given destination (<DESTINATION-IP>:<DESTINATION-PORT>) at most 65536 times concurrently, which is the total number of possible values for <SOURCE-PORT>. Importantly, it’s okay for two connections to use the same source port, provided that they are connecting to a different destination IP and/or port.

    Usually a program will ask Linux (or any other OS) to automatically choose an available source port to satisfy the rules. If no port is available (because 65536 connections to the given destination (<DESTINATION-IP>:<DESTINATION-PORT>) are already open), then the OS will respond with EADDRINUSE.

    This is a little complicated by a feature of TCP called “TIME_WAIT”. When a given connection is closed, the TCP specification declares that both ends should wait a certain amount of time before opening a new connection with the same 4-tuple. This is to avoid the possibility that delayed packets from the first connection might be misconstrued as belonging to the second connection.

    Generally this TIME_WAIT waiting period lasts for only a minute or two. In practice, this means that even if 65536 connections are not currently open to a given destination IP and port, if enough recent connections were open, there still may not be a source port available for use in a new connection. In practice even fewer concurrent connections may be possible since Linux tries to select source ports randomly until it finds an available one, and with enough source ports used up, it may not find a free one before it gives up.

    Port exhaustion in Heroku’s routing layer

    So why would we see EADDRINUSE in connections from the routing layer to dynos? According to our understanding, such an error should not happen. It would indicate that 65536 connections from a specific routing node were being made to a specific dyno. This should mean that the theoretical limit on concurrent connections should be far more than a single dyno could ever hope to handle.

    We could easily see from our application traffic graphs that no dyno was coming close to this theoretical limit. So we were left with a concerning mystery: how was it possible that we were seeing EADDRINUSE errors?

    We wanted to prevent the incident from ever happening again, and so we continued to dig - taking a dive into the internals of our systems.

    Our routing layer is written in Erlang, and the most likely candidate was its virtual machine’s TCP calls. Digging through the VM’s network layer we got down to the sock_connect call which is mostly a portable wrapper around the linux connect() syscall.

    Seeing this, it seemed that nothing in there was out of place to cause the issue. We’d have to go deeper, in the OS itself.

    After digging and reading many documents, one of us noticed this bit in the now well-known blog post Bind before connect:

    Bind is usually called for listening sockets so the kernel needs to make sure that the source address is not shared with anyone else. It's a problem. When using this techique [sic] in this form it's impossible to establish more than 64k (ephemeral port range) outgoing connections in total. After that the attempt to call bind() will fail with an EADDRINUSE error - all the source ports will be busy.


    When we call bind() the kernel knows only the source address we're asking for. We'll inform the kernel of a destination address only when we call connect() later.

    This passage seems to be describing a special case where a client wants to make an outgoing connection with a specific source IP address. We weren’t doing that in our Erlang code, so this still didn’t seem to fit our situation well. But the symptoms matched so well that we decided to check for sure whether the Erlang VM was doing a bind() call without our knowledge.

    We used strace to determine the actual system call sequence being performed. Here’s a snippet of strace output for a connection to

    *bind*(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("")}, 16) = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("")}, 16) = 0

    To our surprise, bind() was being called! The socket was being bound to a <SOURCE-IP>:<SOURCE-PORT> of Why?

    This instructs the kernel to bind the socket to any IP and any port. This seemed a bit useless to us, as the kernel would already select an appropriate <SOURCE-IP> when connect() was called, based on the destination IP address and the routing table.

    This bind() call seemed like a no-op. But critically, this call required the kernel to select the <SOURCE-IP> right then and there, without having any knowledge of the other 3 parts of the 4-tuple: <SOURCE-IP>, <DESTINATION-IP>, and <DESTINATION-PORT>. The kernel would therefore have only 65536 possible choices and might return EADDRINUSE, as per the bind() manpage:

    EADDRINUSE (Internet domain sockets) The port number was specified as zero in the socket address structure, but, upon attempting to bind to an ephemeral port, it was determined that all port numbers in the ephemeral port range are currently in use. See the discussion of /proc/sys/net/ipv4/ip_local_port_range ip(7).

    Unbeknownst to us, we had been operating for a very long time with far lower of a tolerance threshold than expected -- the ephemeral port range was effectively a limit to how much traffic we could tolerate per routing layer instance, while we thought no such limitation existed.

    The Fix

    Reading further in Bind before connect yields the fix: just set the SO_REUSEADDR socket option before the bind() call. In Erlang this is done by simply passing {reuseaddr, true}.

    At this point we thought we had our answer, but we had to be sure. We decided to test it.

    We first wrote a small C program that exercised the current limit:

    #include <sys/types.h>
    #include <sys/socket.h>
    #include <stdio.h>
    #include <string.h>
    #include <arpa/inet.h>
    #include <unistd.h>
    int main(int argc, char **argv) {
      /* usage: ./connect_with_bind <num> <dest1> <dest2> ... <destN>
       * Opens <num> connections to port 80, round-robining between the specified
       * destination IPs.  Then it opens the same number of connections to port
       * 443.
      int i;
      int fds[131072];
      struct sockaddr_in sin;
      struct sockaddr_in dest;
      memset(&sin, 0, sizeof(struct sockaddr_in));
      sin.sin_family = AF_INET;
      sin.sin_port = htons(0);  // source port 0 (kernel picks one)
      sin.sin_addr.s_addr = htonl(INADDR_ANY);  // source IP
      for (i = 0; i < atoi(argv[1]); i++) {
        memset(&dest, 0, sizeof(struct sockaddr_in));
        dest.sin_family = AF_INET;
        dest.sin_port = htons(80);
        // round-robin between the destination IPs specified
        dest.sin_addr.s_addr = inet_addr(argv[2 + i % (argc - 2)]);
        fds[i] = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
        bind(fds[i], (struct sockaddr *)&sin, sizeof(struct sockaddr_in));
        connect(fds[i], (struct sockaddr *)&dest, sizeof(struct sockaddr_in));
      fprintf(stderr, "GOING TO START CONNECTING TO PORT 443\n");
      for (i = 0; i < atoi(argv[1]); i++) {
        memset(&dest, 0, sizeof(struct sockaddr_in));
        dest.sin_family = AF_INET;
        dest.sin_port = htons(443);
        dest.sin_addr.s_addr = inet_addr(argv[2 + i % (argc - 2)]);
        fds[i] = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
        bind(fds[i], (struct sockaddr *)&sin, sizeof(struct sockaddr_in));
        connect(fds[i], (struct sockaddr *)&dest, sizeof(struct sockaddr_in));

    We increased our file descriptor limit and ran this program as follows:

    ./connect_with_bind 65536

    This program attempted to open 65536 connections to port 80 on the three IPs specified. Then it attempted to open another 65536 connections to port 443 on the same IPs. If only the 4-tuple were in play, we should be able to open all of these connections without any problem.

    We ran the program under strace while monitoring ss -s for connection counts. As expected, we began seeing EADDRINUSE errors from bind(). In fact, we saw these errors even before we’d opened 65536 connections. The Linux kernel does source port allocation by randomly selecting a candidate port and then checking the N following ports until it finds an available port. This is an optimization to prevent it from having to scan all 65536 possible ports for each connection.

    Once that baseline was established, we added the SO_REUSEADDR socket option. Here are the changes we made:

    --- connect_with_bind.c 2016-12-22 10:29:45.916723406 -0500
    +++ connect_with_bind_and_reuse.c   2016-12-22 10:31:54.452322757 -0500
    @@ -17,6 +17,7 @@
       int fds[131072];
       struct sockaddr_in sin;
       struct sockaddr_in dest;
    +  int one = 1;
       memset(&sin, 0, sizeof(struct sockaddr_in));
    @@ -33,6 +34,7 @@
         dest.sin_addr.s_addr = inet_addr(argv[2 + i % (argc - 2)]);
         fds[i] = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
    +    setsockopt(fds[i], SOL_SOCKET, SO_REUSEADDR, &one, sizeof(int));
         bind(fds[i], (struct sockaddr *)&sin, sizeof(struct sockaddr_in));
         connect(fds[i], (struct sockaddr *)&dest, sizeof(struct sockaddr_in));
    @@ -48,6 +50,7 @@
         dest.sin_addr.s_addr = inet_addr(argv[2 + i % (argc - 2)]);
         fds[i] = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
    +    setsockopt(fds[i], SOL_SOCKET, SO_REUSEADDR, &one, sizeof(int));
         bind(fds[i], (struct sockaddr *)&sin, sizeof(struct sockaddr_in));
         connect(fds[i], (struct sockaddr *)&dest, sizeof(struct sockaddr_in));

    We ran it like this:

    ./connect_with_bind_and_reuse 65536

    Our expectation was that bind() would stop returning EADDRINUSE. The new program confirmed this fairly rapidly, and showed us once more that what you may expect from theory and practice has quite a gap to be bridged.

    Knowing this, all we had to do is confirm that the {reuseaddr, true} option for the Erlang side would work, and a quick strace of a node performing the call confirmed that the appropriate setsockopt() call was being made.

    Giving Back

    It was quite an eye-opening experience to discover this unexpected connection limitation in our routing layer. The patch to Vegur, our open-sourced HTTP proxy library, was deployed a couple of days later, preventing this issue from ever biting us again.

    We hope that sharing our experience here, we might save you from similar bugs in your systems.

  • N+1 Queries or Memory Problems: Why not Solve Both? (Heroku)
    28 Mar 2017 15:32

    This post is going to help save you money if you're running a Rails server. It starts like this: you write an app. Let's say you're building the next hyper-targeted blogging platform for medium length posts. When you login, you see a paginated list of all of the articles you've written. You have a Post model and maybe for to do tags, you have a Tag model, and for comments, you have a Comment model. You write your view so that it renders the posts:

    <% @posts.each do |post| %>
      <%= link_to(post, post.title) %>
      <%= teaser_for(post) %>
      <%= "#{post.comments.count} comments"
    <% end %>
    <%= pagination(@posts) %>

    See any problems with this? We have to make a single query to return all the posts - that's where the @posts comes from. Say that there are N posts returned. In the code above, as the view iterates over each post, it has to calculate post.comments.count - but that in turn needs another database query. This is the N+1 query problem - our initial single query (the 1 in N+1) returns something (of size N) that we iterate over and perform yet another database query on (N of them).

    Introducing Includes

    If you've been around the Rails track long enough you've probably run into the above scenario before. If you run a Google search, the answer is very simple -- "use includes". The code looks like this:

    # before
    @posts = current_user.posts.per_page(20).page(params[:page])

    and after

    @posts = current_user.posts.per_page(20).page(params[:page])
    @posts = @posts.includes(:comments)

    This is still textbook, but let's look at what's going on. Active Record uses lazy querying so this won't actually get executed until we call @posts.first or @posts.all or @posts.each. When we do that two queries get executed, the first one for posts makes sense:

    select * from posts where user_id=? limit ? offset ?

    Active Record will pass in user_id and limit and offset into the bind params and you'll get your array of posts.

    Note: we almost always want all queries to be scoped with a limit in production apps.

    The next query you'll see may look something like this:

    select * from comments where post_id in ?

    Notice anything wrong? Bonus points if you found it, and yes, it has something to do with memory.

    If each of those 20 blog posts has 100 comments, then this query will return 2,000 rows from your database. Active Record doesn't know what data you need from each post comment, it'll just know it was told you'll eventually need them. So what does it do? It creates 2,000 Active Record objects in memory because that's what you told it to do. That's the problem, you don't need 2,000 objects in memory. You don't even need the objects, you only need the count.

    The good: You got rid of your N+1 problem.

    The bad: You're stuffing 2,000 (or more) objects from the database into memory when you aren't going to use them at all. This will slow down this action and balloon the memory use requirements of your app.

    It's even worse if the data in the comments is large. For instance, maybe there is no max size for a comment field and people write thousand word essays, meaning we'll have to load those really large strings into memory and keep them there until the end of the request even though we're not using them.

    N+1 Is Bad, Unneeded Memory Allocation Is Worse

    Now we've got a problem. We could "fix" it by re-introducing our N+1 bug. That's a valid fix, however, you can easily benchmark it. Use rack-mini-profiler in development on a page with a large amount of simulated data. Sometimes it's faster to not "fix" your N+1 bugs.

    That's not good enough for us, though -- we want no massive memory allocation spikes and no N+1 queries.

    Counter Cache

    What's the point of having Cache if you can't count it? Instead of having to call post.comments.count each time, which costs us a SQL query, we can store that data directly inside of the Post model. This way when we load a Post object we automatically have this info. From the docs for the counter cache you'll see we need to change our model to something like this:

    class Comment < ApplicationRecord
       belongs_to :post , counter_cache: count_of_comments

    Now in our view, we can call:

      <%= "#{post.count_of_comments} comments"  %>

    Boom! Now we have no N+1 query and no memory problems. But...

    Counter Cache Edge Cases

    You cannot use a counter cache with a condition. Let's change our example for a minute. Let's say each comment could either be "approved", meaning you moderated it and allow it to show on your page, or "pending". Perhaps this is a vital piece of information and you MUST show it on your page. Previously we would have done this:

      <%= "#{ post.comments.approved.count } approved comments"  %>
      <%= "#{ post.comments.pending.count } pending comments"  %>

    In this case the Comment model has a status field and calling comments.pending is equivalent to adding where(status: "pending"). It would be great if we could have a post.count_of_pending_comments cache and a post.count_of_approved_comments cache, but we can't. There are some ways to hack it, but there are edge cases, and not all apps can safely accommodate for all edge cases. Let's say ours is one of those.

    Now what? We could get around this with some view caching because if we cache your entire page, we only have to render it and pay that N+1 cost once. Maybe fewer times if we are re-using view components and are using "Russian doll" style view caches .

    If view caching is out of the question due to <reasons>, what are we left with? We have to use our database the way the original settlers of the Wild West did, manually and with great effort.

    Manually Building Count Data in Hashes

    In our controller where we previously had this:

    @posts = current_user.posts.per_page(20).page(params[:page])
    @posts = @posts.includes(:comments)

    We can remove that includes and instead build two hashes. Active Record returns hashes when we use group(). In this case we know we want to associate comment count with each post, so we group by :post_id.

    @posts = current_user.posts.per_page(20).page(params[:page])
    post_ids =
    @pending_count_hash   = Comment.pending.where(post_id: post_ids).group(:post_id).count
    @approved_count_hash = Comment.approved.where(post_id: post_ids).group(:post_id).count

    Now we can stash and use this value in our view instead:

      <%= "#{ @approved_count_hash[] || 0  } approved comments"  %>
      <%= "#{ @pending_count_hash[] || 0 } pending comments"  %>

    Now we have 3 queries, one to find our posts and one for each comment type we care about. This generates 2 extra hashes that hold the minimum of information that we need.

    I've found this strategy to be super effective in mitigating memory issues while not sacrificing on the N+1 front.

    But what if you're using that data inside of methods.

    Fat Models, Low Memory

    Rails encourage you to stick logic inside of models. If you're doing that, then perhaps this code wasn't a raw SQL query inside of the view but was instead nested in a method.

    def approved_comment_count

    Or maybe you need to do the math, maybe there is a critical threshold where pending comments overtake approved:

    def comments_critical_threshold?
      self.comments.pending.count < self.comments.approved.count

    This is trivial, but you could imagine a more complex case where logic is happening based on business rules. In this case, you don't want to have to duplicate the logic in your view (where we are using a hash) and in your model (where we are querying the database). Instead, you can use dependency injection. Which is the hyper-nerd way of saying we'll pass in values. We can change the method signature to something like this:

    def comments_critical_threshold?(pending_count: comments.pending.count, approved_count: comments.approved.count)
      pending_count < approved_count

    Now I can call it and pass in values:

    post.comments_critical_threshold?(pending_count: @pending_count_hash[] || 0 , approved_count: @approved_count_hash[] || 0 )

    Or, if you're using it somewhere else, you can use it without passing in values since we specified our default values for the keyword arguments.

    BTW, aren't keyword arguments great?

    post.comments_critical_threshold? # default values are used here

    There are other ways to write the same code:

    def comments_critical_threshold?(pending_count , approved_count )
      pending_count ||= comments.pending.count
      approved_count ||= comments.approved.count
      pending_count < approved_count

    You get the gist though -- pass values into your methods if you need to.

    More than Count

    What if you're doing more than just counting? Well, you can pull that data and group it in the same way by using select and specifying multiple fields. To keep going with our same example, maybe we want to show a truncated list of all commenter names and their avatar URLs:

    @comment_names_hash = Comment.where(post_id: post_ids).select("names, avatar_url").group_by(&:post_ids)

    The results look like this:

    1337: [
      { name: "schneems", avatar_url: "" },
      { name: "illegitimate45", avatar_url: "" }

    The 1337 is the post id, and then we get an entry with a name and an avatar_url for each comment. Be careful here, though, as we're returning more data-- you still might not need all of it and making 2,000 hashes isn't much better than making 2,000 unused Active Record objects. You may want to better constrain your query with limits or by querying for more specific information.

    Are We There Yet

    At this point, we have gotten rid of our N+1 queries and we're hardly using any memory compared to before. Yay! Self-five. :partyparrot:. 🎉

    Here's where I give rapid-fire suggestions.

    • Use the bullet gem -- it will help identify N+1 query locations and unused includes -- it's good.
    • Use rack-mini-profiler in development. This will help you compare relative speeds of your performance work. I usually do all my perf work on a branch and then I can easily go back and forth between that and master to compare speeds.
    • Use production-like data in development. This performance "bug" won't show until we've got plenty of posts or plenty of comments. If your prod data isn't sensitive you can clone it using something like $ heroku pg:pull to test against, but make sure you're not sending out emails or spending real money or anything first.
    • You can see memory allocations by using rack-mini-profiler with memory-profiler and adding pp=profile-memory to the end of your URL. This will show you things like total bytes allocated, which you can use for comparison purposes.
    • Narrow down your search by focusing on slow endpoints. All performance trackers list out slow endpoints, this is a good place to start. Scout will show you memory breakdown per request and makes finding these types of bugs much easier to hunt down. They also have an add-on for Heroku. You can get started for free $ heroku addons:create scout:chair

    If you want to dig deeper into what's going on with Ruby's use of memory check out the Memory Quota Exceeded in Ruby (MRI) Dev Center article , my How Ruby Uses Memory, and also Nate Berkopec's Halve your memory use with these 12 Weird Tricks.

  • Announcing Free and Automated SSL Certs For All Paid Dynos (Heroku)
    21 Mar 2017 15:43

    We are happy to announce the general availability of Automated Certificate Management (ACM) for all paid Heroku dynos. With ACM, the cumbersome and costly process of provisioning and managing SSL certificates is replaced with a simple experience that is free for all paid Dynos on Heroku’s Common Runtime. Creating secure web applications has never been more important, and with ACM and the Let’s Encrypt project, never easier.

    ACM handles all aspects of SSL/TLS certificates for custom domains; you no longer have to purchase certificates, or worry about their expiration or renewal. ACM builds directly on our recent release of Heroku Free SSL to make encryption the default for web applications and helps you protect against eavesdropping, cookie theft, and content hijacking. Heroku has always made it easy to add SSL encryption to web applications — today’s release of ACM extends that further to automatically generate a TLS certificate issued by Let’s Encrypt for your application’s custom domains.

    How It Works

    New Applications

    Every time you upgrade from a Free dyno to a Hobby or Professional dyno, we will automatically generate a TLS certificate for all custom domains on your application. You will need to ensure that your application’s custom domains are pointed to the correct DNS targets as specified in heroku domains.

    Existing Applications

    For existing applications, you can enable ACM by simply going to your application’s settings page and clicking the “Configure SSL” button.

    Or you can run the CLI command:

    $ heroku certs:auto:enable -a <app name>

    If your application was not using Heroku SSL, update your DNS settings for your custom domain to its new DNS target and run heroku domains to verify.

    Run the following to verify whether your application’s domains are covered by Automated Certificate Management:

    $ heroku certs:auto

    For more details, including how to migrate from the SSL endpoint add-on, please see our Dev Center documentation.

  • Introducing the Einstein Vision Add-on for Image Recognition (Heroku)
    07 Mar 2017 13:00

    The most innovative apps augment our human senses, intuition, and logic with machine learning. Deep learning, modelled after the neural networks of the human brain, continues to grow as one of the most powerful types of machine learning. When applied to images, deep learning enables powerful computer vision features like visual search, product identification, and brand detection.

    Today, we bring you the Einstein Vision add-on (beta), allowing Heroku developers to easily connect to and use Einstein Vision, a set of powerful new APIs for building AI-powered apps. With this release, Salesforce is making it easy for you to embed image recognition directly into your apps. Rather than building and managing the specialized infrastructure needed to host deep learning models, simply connect to Einstein Vision's HTTP/REST API for custom image recognition with little development overhead.

    Use Einstein Vision to discover your products across your social media channels, analyze observational data in healthcare and life science applications, and enable visual search in eCommerce apps to delight your customers. Get started quickly with pre-trained image classifiers that are automatically available to you when you install the Einstein Vision add-on.

    A Simple Workflow for Custom Image Recognition

    The true strength of Einstein Vision is its ability to train custom models to recognize the things you care about. Creating a custom model takes just a few steps:

    1. Plan
    2. Collect
    3. Train & Evaluate
    4. Query

    Let's walk through the workflow to create a brand recognizer for Heroku logos and artwork, which is based on the Einstein Vision example app in Node.js.

    Plan the Model: Label All the Things

    In machine learning, the “model” is the brain that answers questions, and “labels” are the possible answers. To have Einstein Vision recognize specific objects, we will train the model using example images for each label. For the example brand recognizer app, labels represent visual aspects of the Heroku brand.

    Start with the labels of primary interest:

    • Heroku logo, isolated logos
    • Heroku artwork, various supporting imagery
    • Heroku swag, t-shirts, socks, water bottles, etc.

    Then, think about images that do not contain one of the objects we want to recognize. How will the model answer those questions? Let's plan a negative, catch-all label representing the infinite world of objects beyond our target labels:

    • Unknown, a random set of things we don't care about

    The unknown set is a curiosity at first. Remember that the model can only answer questions it's been trained to answer. If you want a clear indication of the model not matching a label, then train negative labels as well.

    Collect Example Images

    Before diving into the actual machine learning, we must gather example images that represent each of the planned labels. Each label needs a variety of example images: in isolation, in normal surroundings, from various angles, with various compositions, and with 2D & 3D representations. This avoids over-fitting the model, improving the flexibility in classification of unseen images. We collect examples by sorting them into a directory named for each label, preparing them for zip upload into a dataset.

    While a model can be trained with very few images per label, more training examples will dramatically improve prediction accuracy for unseen images. We've built demos with just a few dozen examples per label, but at least a thousand images per label is recommended for high-confidence predictions.

    Train The Model

    Once the example images are collected, we will use the REST/HTTP API provided by the add-on to upload the dataset.

    The steps to train a model are:

    1. Upload example images
    2. Initiate training
    3. Check training status
    4. Inspect the model's metrics

    Walkthrough the API flow with our example.

    Performance Evaluation

    After training, Einstein Vision automatically evaluates the model using cross-validation. It withholds a random 10% (k-fold = 10) of the example data to test the model. The accuracy of predictions from that portion of unseen data, the testAccuracy, represents how the model will perform in the real-world.

    Fetch model metrics from the API for any trained model to get its testAccuracy. Additional metrics returned may indicate issues with examples confusing the algorithm or errors reducing the useful dataset.

    To tune a model, revise the source dataset to address any issues and then create, train, and evaluate a new model. After tuning, the model with superior metrics may be considered production-ready.

    Query The Model

    Once training is complete, the new model will answer queries to classify images by URL reference or direct upload. Here's an example query using the curl command-line tool:

    $ curl -X POST \
      -F "sampleContent=@./path/to/image.jpg" \
      -F "modelId=YYYYY" \
      -H "Authorization: Bearer XXXXX" \
      -H "Content-Type: multipart/form-data" \

    Example JSON response:

      "probabilities": [
          "label": "Heroku Artwork",
          "probability": 0.53223926
          "label": "unknown",
          "probability": 0.46305126
          "label": "Heroku Swag",
          "probability": 0.0038324401
          "label": "Heroku Logo",
          "probability": 0.0008770062
      "object": "predictresponse"

    Pipeline to Production

    One of the wonderful things about Heroku apps is that after you get a proof-of-concept running, add the app to a Pipeline to enable enterprise-grade continuous delivery, including: Review Apps, CI Tests (in private beta), and elegant promotion to production.

    To share a model created in one app with other apps in a Heroku Pipeline, such as promoting an app from Review App to Staging and finally to Production, the add-on must be shared between those apps.

    Only the Beginning

    We can’t wait to see what you build with the Einstein Vision add-on (beta). Einstein Vision is free to get started, and we plan to introduce paid plans at GA in a few weeks. Check out the add-on documentation, then dive in with our Node example app or add it to your own app to try it out.

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>