My two cents about software development on the web

Cloud Platforms

Warning: file_put_contents(C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache/0bac12a71080e2f0238f1b173488ab6f.spc) [function.file-put-contents]: failed to open stream: Permission denied in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 8680

Warning: C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache is not writeable. Make sure you've set the correct relative or absolute path, and that the location is server-writable. in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 1781

Warning: file_put_contents(C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache/ad31bb8bf02bbf58903d4663ca4ee574.spc) [function.file-put-contents]: failed to open stream: Permission denied in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 8680

Warning: C:\hostingspaces\neetpiqc\\wwwroot/wp-content/cache is not writeable. Make sure you've set the correct relative or absolute path, and that the location is server-writable. in C:\hostingspaces\neetpiqc\\wwwroot\wp-includes\class-simplepie.php on line 1781
  • Windows Server 2012 R2, IIS 8.5, WebSockets and .NET 4.5.2 (AppHarbor Blog)

    During the last couple of weeks we've upgraded worker servers in the US and EU regions to support Windows Server 2012 R2, IIS 8.5 and .NET 4.5.2. Major upgrades like this can be a risky and lead to compatibility issues, and the upgrade was carefully planned and executed to maximize compatibility with running applications. Application performance and error rates have been closely monitored throughout the process and fortunately, chances are you haven't noticed a thing: We've detected migration-related issues with less than 0.1% of running applications.

    Many of the new features and configuration improvements enabled by this upgrade will be gradually introduced over the coming months. This way we can ensure a continued painless migration and maintain compatibility with the previous Windows Server 2008 R2/IIS 7.5 setup, while we iron out any unexpected kinks if and when they crop up. A few changes have however already been deployed that we wanted to fill you in on.

    WebSocket support and the beta region

    Last year the beta region featuring experimental WS2012 and WebSockets support was introduced. The beta region allowed customers to test existing and new apps on the new setup while we prepared and optimized it for production use. This approach has been an important factor in learning about subtle differences between the server versions, and addressing pretty much all compatibility issues before upgrading the production regions. Thanks to all the customers who provided valuable feedback during the beta and helped ensure a smoother transition for everyone.

    An important reason for the server upgrade was to support WebSocket connections. Now that the worker servers are running WS2012 and IIS 8.5 we've started doing just that. Applications in the old beta region have been merged into the production US region and the beta region is no longer available when you create a new application.

    Most load balancers already support WebSockets and the upgrade is currently being rolled out to remaining load balancers. Apps created since August 14th fully support WebSockets and no configuration is necessary: AppHarbor will simply detect and proxy connections as expected when a client requests a Connection: Upgrade.

    Some libraries, such as SignalR, will automatically detect and prefer WebSocket connections when supported by both the server and client. Until WebSocket connections are supported on all load balancers some apps may attempt and fail during the WebSocket handshake. This should not cause issues since these libraries will fall back to other supported transports, and affected apps will automatically be WebSocket-enabled when supported by the load balancers.

    CPU throttling

    One of the major challenges that has held back this upgrade is a change in the way we throttle worker CPU usage. CPU limitations are the same as before, but the change can affect how certain CPU-intensive tasks are executed. Resources and documentation on this subject are limited, but testing shows that CPU time is more evenly scheduled across threads, leading to higher concurrency, consistency and stability within processes. While this is overall an improvement it can also affect peak performance on individual threads, and we're currently investigating various approaches to better support workloads affected by this.

    For the curious, we previously used a CPU rate limit registry setting to limit CPU usage per user account, but this is no longer supported on Windows Server 2012. We now use a combination of IIS 8's built-in CPU throttling and a new CPU rate control for job objects to throttle background workers.

    If you've experienced any issues with this upgrade or have feedback about the process, please don't hesitate to reach out.

  • Heartbleed Security Update (AppHarbor Blog)

    Updated on April 10, 2014 with further precautionary steps in the "What can you do" section below.

    On April 7, 2014, a serious vulnerability in the OpenSSL library (CVE-2014-0160) was publicly disclosed. OpenSSL is a cryptography library used for the majority of private communications across the internet.

    The vulnerability, nicknamed "Heartbleed", would allow an attacker to steal secret certificates keys, names and passwords of users and other secrets encrypted using the OpenSSL library. As such it represents a major risk for a large number of internet application and services, including AppHarbor.

    What has AppHarbor done about this

    AppHarbor responded to the announcement by immediately taking steps to remediate the vulnerability:

    1. We updated all affected components with the updated, secure version of OpenSSL within the first few hours of the announcement. This included SSL endpoints and load balancers, as well as other infrastructure components used internally at AppHarbor.
    2. We re-keyed and redeployed all potentially affected AppHarbor SSL certificates (including the piggyback * certificate), and the old certificates are being revoked.
    3. We notified customers with custom SSL certificates last night, so they could take steps to re-key and reissue certificates, and have the old ones revoked.
    4. We reset internal credentials and passwords.
    5. User session cookies were revoked, requiring all users to sign in again.

    Furthermore, AppHarbor validates session cookies against your previously known IP addresses as part of the authorization process. This has reduced the risk of a stolen session cookie being abused. Perfect forward secrecy was deployed to some load balancers, making it impossible to read intercepted and encrypted communication with stolen keys. Forward secrecy has since been deployed to all load balancers hosted by AppHarbor.

    What can you do

    We have found no indication that the vulnerability was used to attack AppHarbor. By quickly responding to the issue and taking the steps mentioned above we effectively stopped any further risk of exposure. However, due to the nature of this bug, we recommend users who want to be extra cautious to take the following steps:

    1. Reset your AppHarbor password.
    2. Review the sign-in and activity history on your user page for any suspicious activity.
    3. Revoke authorizations for external applications that integrates with AppHarbor.
    4. Recreate, reissue and reinstall custom SSL certificates you may have installed, and revoke the old ones. Doing this may revoke the old certificates, so make sure you're ready to install the new certificates.
    5. Read the details about the Heartbleed bug here and assess the risks relative to your content.

    Updated instructions (April 10, 2014):

    While we still have not seen any abuse on AppHarbor as a result of this bug, we now also encourage you to take these precautionary steps:

    1. Reset your build URL token.
    2. If you're using one of the SQL Server or MySQL add-ons: Reset the database password. Go to the add-on's admin page and click the "Reset Password" button. This will immediately update the configuration on AppHarbor and redeploy the application (with a short period of downtime until it is redeployed).
    3. If you're using the Memcacher add-on: Reinstall the add-on by uninstalling and installing it.
    4. Rotate/update sensitive information in your own configuration variables.

    If you have hardcoded passwords/connection strings for any your add-ons this is a good opportunity to start using the injected configuration variables. You can find instructions for the SQL add-ons here and the Memcacher add-on here. This way your application is automatically updated when you reset the add-ons, or when an add-on provider updates the configuration. If this is not an option you should immediately update your code/configuration files and redeploy the application after the configuration is updated.

    Stay tuned

    Protecting your code and data is our top priority, and we continue to remediate and asses the risks in response to this issue. We'll keep you posted with any new developments, so stay tuned on Twitter and the blog for important updates. We're of course also standing by on the support forums if you have any questions or concerns.

  • Librato integration and built-in perfomance metrics (AppHarbor Blog)

    Librato Dashboard

    Being able to monitor and analyze key application metrics is an essential part of developing stable, performant and high quality web services that meets you business requirements. Today we’re announcing a great new set of features to provide a turnkey solution for visualizing, analyzing and acting on key performance metrics. On top of that we’re enabling you to easily track your own operational metrics. In this blog post we’ll look at how the pieces tie together.

    Librato integration

    The best part of today’s release is our new integration with Librato for monitoring and analyzing metrics. Librato is an awesome and incredibly useful service that enables you to easily visualize and correlate metrics, including the new log-based performance metrics provided by AppHarbor (described in more details below).

    Librato Dashboard

    Librato is now available as an add-on and integrates seamlessly with your AppHarbor logs. When you provision the add-on, Librato will setup a preconfigured dashboard tailored for displaying AppHarbor performance data and you can access it immediately by going to the Librato admin page. Everything will work out of the box without any further configuration and your logs will automatically be sent to Librato using a log drain.

    When log messages containing metric data are sent to Librato they’re transformed by an l2met service before being sent to their regular API. A very cool feature of the l2met service is that it can automatically calculate some useful metrics. For instance, it’ll calculate the median response time as well as the the 99th and 95th percentile of measurements such as response times. The perc99 response time means the response time of the 99% fastest responses. It can be useful to know this value since it's less affected by a few very slow responses than the average. Among other things this provides a good measurement of the browsing experience for most of your users.

    Librato Dashboard

    The l2met project was started by Ryan Smith - a big shout-out and thanks to him and the Librato team for developing this great tool.

    For more information about how to integrate with Librato and details about the service please refer to the documentation here. Also check out their announcement blog post about the integration.

    Built-in performance metrics

    AppHarbor can now write key runtime performance metrics directly to your application’s log stream as l2met 2.0 formatted messages similar to this:

    source=web.5 sample#memory.private_bytes=701091840
    source=web.5 sample#process.handles=2597
    source=web.5 sample#cpu.load_average=1.97

    These are the messages Librato uses as well and most of them are written every 20 seconds. They allow for real-time monitoring of worker-specific runtime metrics such as CPU (load average) and memory usage, as well as measurements of response time and size reported from the load balancers. Because these metrics are logged to your log stream you can also consume them in the same way you’d usually view or integrate with your logs.

    Load average run-time metrics

    Performance data collection takes place completely out-of-process, without using a profiler, and it can be enabled and disabled without redeploying the application. This means that monitoring won’t impact application performance at all and that a profiler (such as New Relic) can still be attached to the application.

    Writing custom metrics

    The performance data provided by AppHarbor is probably not the only metrics you want to track. You can of course integrate directly with Librato’s API, but the l2met integration makes it easier than ever to track your own metrics, and the paid Librato plans includes the ability to track custom metrics exactly for that purpose.

    You can start writing your own metrics simply by sending an l2met-formatted string to your logs. Last week we introduced the Trace Logging feature which is perfect for this, so writing your custom metrics can now be done with a simple trace:


    To make this even easier we’ve built the metric-reporter library (a .NET port of Librato’s log-reporter) to provide an easy to use interface for writing metrics to your log stream. You can install it with NuGet:

    Install-Package MetricReporter

    Then initialize a MetricReporter which writes to a text writer:

    var writer = new L2MetWriter(new TraceTextWriter);
    var reporter = new MetricReporter(metricWriter);

    And start tracking your own custom metrics:

    reporter.Measure("payload.size", 21276);
    reporter.Measure("twitter.lookup.time", () =>
        //Do work

    On Librato you can then view charts with these new metrics along with the performance metrics provided by AppHarbor, and add them to your dashboards, aggregate and correlate data, set up alerts etc. The MetricReporter library will take care of writing l2met-formatted metrics using the appropriate metric types and write to the trace or another IO stream. Make sure to inspect the README for more examples and information on configuration and usage.

    That’s all we have for today. There’ll be more examples on how you can use these new features soon, but for now we encourage you to take it for a spin, install the Librato add-on and test the waters for yourself. We’d love to hear what you think so if there are other metrics you’d like to see or if you experience any issues please hit us up through the usual channels.

  • Introducing Trace Logging (AppHarbor Blog)

    Today we’re happy to introduce trace message integration with your application log. With tracing you can very easily log trace messages to your application's log stream by using the built-in tracing capabilities of the .NET framework from anywhere in your application.

    When introducing the realtime logging module a while back we opened up access to collated log data from load balancers, the build and deploy infrastructure, background workers and more. Notably missing however was the ability to log from web workers. We’re closing that gap with tracing, which can be used in both background and web workers.

    How to use it

    The trace feature integrates with standard .NET tracing, so you don’t have to make any changes to your application to use it. You can simply log traces from your workers with the System.Diagnostics.Trace class:

    Trace.TraceInformation("Hello world");

    This will yield a log message containing a timestamp and the source of the trace in your application’s log like so:

    2014-01-22T06:46:48.086+00:00 app web.1 Hello World

    You can also use a TraceSource by specifying the trace source name AppHarborTraceSource:

    var traceSource = new TraceSource("AppHarborTraceSource", defaultLevel: SourceLevels.All);
    traceSource.TraceEvent(TraceEventType.Critical, 0, "Foo");

    You may not always want noisy trace messages in your logs and you can configure the trace level on the "Logging" page. There are 4 levels: All, Warning, Error and None. Setting the trace level will update the configuration without redeploying or restarting the application. This is often desirable if you need to turn on tracing when debugging and diagnosing an ongoing or state-related issue.

    Configure Trace level

    There are a number of other ways to use the new tracing feature including:

    • ASP.NET health monitoring (for logging exceptions, application lifecycle events etc).
    • A logging library such as NLog (Trace target) or log4net (TraceAppender).
    • Integrating with ETW (Event Tracing for Windows) directly using the injected event provider id.

    Anything that integrates with .NET tracing or ETW should work, and you can find more details and examples in this knowledge base article.

    All new applications have tracing enabled by default. Tracing can be enabled for existing applications on the "Logging" page.

    How does it work

    Under the hood we’re using ETW for delivering log messages to the components that are responsible for sending traces to your log stream. Application performance is unaffected by the delivery of log messages as this takes place completely out of process. Note however that messages are buffered for about a second and that some messages may be dropped if you’re writing excessively to the trace output.

    When tracing is enabled, AppHarbor configures your application with an EventProviderTraceListener as a default trace listener. While you can integrate directly with ETW as well we recommend using the Trace or TraceSource approaches described above.

    Viewing trace messages

    Traces are collated with other logging sources in your log stream, so you can consume them in the same way you’re used to. You can view log messages using the command line interface, the web viewer or set up a log drain to any HTTP, HTTPS or syslog endpoint. For more information about the various integration points please refer to this article.

    Viewing trace messages in console

    We’ve got a couple of cool features that builds on this ready soon, so stay tuned and happy tracing!

  • .NET 4.5.1 is ready (AppHarbor Blog)

    Microsoft released .NET 4.5.1 a while back, bringing a bunch of performance improvements and new features to the framework. Check out the announcement for the details.

    Over the past few weeks we have updated our build infrastructure and application servers to support this release. We're happy to report that AppHarbor now supports building, testing and running applications targeting the .NET 4.5.1 framework, as well as solutions created with Visual Studio 2013 and ASP.NET MVC 5 applications.

    There are no known issues related to this release. If you encounter problems, please refer to the usual support channels and we'll help you out.

    .NET logo

  • Integrated NuGet Package Restore (AppHarbor Blog)

    A few months ago the NuGet team released NuGet 2.7, which introduced a new approach to package restore. We recently updated the AppHarbor build process to adopt this approach and integrate the new NuGet restore command. AppHarbor will now automatically invoke package restore before building your solution.

    Automatically restoring packages is a recommended practice, especially because you don’t have to commit the packages to your repository and can keep the footprint small. Until now we’ve recommended using the approach desribed in this blog post to restore NuGet packages when building your application. This has worked relatively well, but it’s also a bit of a hack and has a few caveats:

    • Some NuGet packages rely files that needs to be present and imported when MSBuild is invoked. This has most notably been an issue for applications relying on the Microsoft.Bcl.Build package for the reasons outlined in this article.
    • NuGet.exe has to be committed and maintained with the repository and project and solution files needs to be configured.
    • Package restore can intermittently fail in some cases when multiple projects are built concurrently.

    With this release we expect to eliminate these issues and provide a more stable, efficient and streamlined way of handling package restore.

    If necessary, NuGet can be configured by adding a NuGet.config file in the same directory as your solution file (or alternatively in a .nuget folder under your solution directory). You usually don't have to configure anything if you’re only using the official NuGet feed, but you’ll need to configure your application if it relies on other package sources. You can find an example configuration file which adds a private package source in the knowledge base article about package restore and further documentation for NuGet configuration files can be found here.

    If you hit any snags we’re always happy to help on our support forums.

    NuGet logo

  • New Relic Improves Service and Reduces Price (AppHarbor Blog)

    New Relic

    We're happy to announce that New Relic has dropped the price of the Professional add-on plan from $45/month to $19/month per worker unit. Over the years New Relic has proven to be a really useful tool for many of our customers, and we're pleased that this price drop will make the features of New Relic Professional more accessible to everyone using AppHarbor.

    Highlights of the Professional plan include:

    • Unlimited data retention
    • Real User Monitoring (RUM) and browser transaction tracing
    • Application transaction tracing, including Key Transactions and Cross Application Tracing
    • Advanced SQL and slow SQL analysis

    You can find more information about the benefits of New Relic Pro on the New Relic website (

    Service update

    The New Relic agent was recently upgraded to a newer version which brings support for some recently introduced features as well as a bunch of bug fixes. Time spent in the request queue is now reported and exposed directly in the New Relic interface. Requests are rarely queued for longer than a few milliseconds, but it can happen if your workers are under load. When more time is spent in the request queue it may be an indicator that you need to scale your application to handle the load efficiently.

    We're also making a few changes to the way the New Relic profiler is initialized with your applications. This is particularly relevant if you've subscribed to New Relic directly rather than installing the add-on through AppHarbor. Going forward you'll need to add a NewRelic.LicenseKey configuration variable to make sure the profiler is attached to your application. We recommend that you make this change as soon as possible. If you're subscribed to the add-on through AppHarbor no action is required and the service will continue to work as usual.

  • Found Elasticsearch add-on available (AppHarbor Blog)

    Found ElasticSearch

    Found provides fully hosted and managed Elasticsearch clusters; each cluster has reserved memory and storage ensuring predictable performance. The HTTPS API is developer-friendly and existing Elasticsearch libraries such as NEST, Tire, PyES and others work out of the box. The Elasticsearch API is unmodified, so for those with an existing Elasticsearch integration it is easy to get started.

    For production and mission critical environments customers can opt for replication and automatic failover to a secondary site, protecting the cluster against unplanned downtime. Security has a strong focus: communication to and from the service is securely transmitted over HTTPS (SSL) and data is stored behind multiple firewalls and proxies. Clusters run in isolated containers (LXC) and customisable ACLs allow for restricting access to trusted people and hosts.

    In the event of a datacenter failure, search clusters are automatically failed over to a working datacenter or, in case of a catastrophic event, completely rebuilt from backup.

    Co-founder Alex Brasetvik says: "Found provides a solution for companies who are keen to use Elasticsearch but not overly keen to spend their time and money on herding servers! We provide our customers with complete cluster control: they can scale their clusters up or down at any time, according to their immediate needs. It's effortless and there's zero downtime."

    More information and price plans are available on the add-on page.

  • Introducing Realtime Logging (AppHarbor Blog)

    Today we're incredibly excited to announce the public beta of our brand new logging module. Starting immediately all new applications created on AppHarbor will have logging enabled. You can enable it for your existing apps on the new "Logging" page.

    We know all too well that running applications on a PaaS like AppHarbor sometimes can feel like a black box. So far we haven't had a unified, simple and efficient way to collate, present and distribute log events from the platform and your apps.

    That's exactly what we wanted to address with our logging solution, and based on the amazing feedback from private beta users we feel confident that you'll find it useful for getting insight about your application and AppHarbor. A big thanks to all the beta testers who have helped us refine and test these new features.

    The new logging module collates log messages from multiple sources, including almost all AppHarbor infrastructure component and your applications - API changes, load balancer request logs, build, deploy and stdout/stderr from your background workers and more can now be accessed and sent to external services in real time.

    Captain's log Consider yourself lucky we're not that much into skeuomorphism


    We're providing two interfaces "out of the box" - a convenient web-interface can be accessed on the Logging page and a new log command has been added to the CLI. [Get the installer directly from here or install with Chocolatey cinst appharborcli.install. To start a "tailing" log session with the CLI, you can for instance run appharbor log -t -s appharbor. Type appharbor log -h to see all options. Log web interface

    The web interface works a bit differently, but try it out and let us know what you think - it's heavily inspired by the project who have built a great client side interface for viewing, filtering, searching and splitting logs into multiple "screens".

    Log web interface


    One of the most useful and interesting aspects of today's release is the flexible integration points it provides. Providing access to your logs in realtime is one thing, but AppHarbor will only store the last 1500 log messages for your application. Storing, searching, viewing and indexing logs can be fairly complex and luckily many services already exists that helps you make more sense of your log data.

    We've worked with Logentries to provide a completely automated and convenient way for sending AppHarbor logs to them when you add their add-on. When you add the Logentries add-on your application can automatically be configured to send logs to Logentries, and Logentries will be configured to display log messages in AppHarbor's format.

    Logentries integration

    You can also configure any syslog (TCP), HTTP and HTTPS endpoint you like with log "drains". You can use this to integrate with services like Loggly and Splunk, or even your own syslog server or HTTP service. More details about log drains are available in the this knowledge base article and the drain API documentation.

    Finally there's a new new Log session API endpoint that you can use to create sessions similar to the ones used by the interfaces we provide.


    If you've ever used Heroku you'll find most of these features very familiar. That's no coincidence - the backend is based on Heroku's awesome distributed syslog router, Logplex. Integrating with Logplex makes it a lot easier for add-on providers who already support Heroku's Logplex to integrate with AppHarbor, while giving us a scalable and proven logging backend to support thousands of deployed apps.

    Logplex is also in rapid, active development, and a big shout-out to the awesome people at Heroku who are building this incredibly elegant solution. If you're interested in learning more about Logplex we encourage you to check out the project on Github and try it for yourself. We've built a client library for interacting with Logplex's HTTP API and HTTP log endpoints from .NET apps - let us know if you'd like to use this and we'll be happy to open source the code. The Logplex documentation on stream management is also useful for a high-level overview of how Logplex works.

    Next steps

    With this release we've greatly improved the logging experience for our customers. We're releasing this public beta since we know it'll be useful to many of you as it is, but we're by no means finished. We want to add even more log sources, provide more information from the various infrastructure components and integrate with more add-on providers. Also note that request logs are currently only available on shared load balancers, but it will be rolled out to all load balancers soon. If you find yourself wanting some log data that is not currently available please let us know. We now have a solid foundation to provide you with the information you need when you need it, and we couldn't be more excited about that.

    We'll provide you with some examples and more documentation for these new features over the next couple of weeks, but for now we hope you'll take it for a spin and test the waters for yourself. Have fun!

  • Introducing PageSpeed optimizations (AppHarbor Blog)

    Today we've introducing a new experimental feature: Google PageSpeed optimizations support. The PageSpeed module is a suite of tools that tries to optimize web page latency and bandwidth usage of your websites by rewriting your content to implement web performance best practices. Reducing the number of requests to a single domain, optimizing cache policies and compressing content can significantly improve web performance and lead to a better user experience.

    With PageSpeed optimization filters we're making it easier to apply some of these best practices, and provide a solution that efficiently and effortlessly speed up your web apps. The optimizations takes place at the load balancer level and works for all web applications no matter what framework or language you use.

    As an example of how this works you can inspect the HTML and resources of this blog to see some of the optimizations that are applied. Analyzing with the online PageSpeed insights tool yields a "PageSpeed score" of 88 when enabled versus 73 when disabled. Not too bad considering it only took a click to enable it.

    PageSpeed button

    You can enable PageSpeed optimizations for your web application on the new "Labs" page, which can be found in the application navigation bar. The application will be configured with PageSpeed's core set of filters within a few seconds. We will then, among other things, apply these filters to your content:

    When you've enabled PageSpeed we recommend that you test the application to make sure it doesn't break anything. You can also inspect the returned content in your browser and if you hit any snags simply disable PageSpeed and let support know about it. Note that only content transferred over HTTP from your domain will be processed by PageSpeed filters. To optimize HTTPS traffic you can enable SPDY support (although that is currently only enabled on dedicated load balancers and in the beta region).

    We'll make more filters available later on, but for the beta we're starting out with a curated set of core filters, which are considered safe for most web applications. There are a few other cool filters we'll add support for later on - such as automatic sprite image generation and lazy-loading of images. Let us know if there are any filters in the catalog you think we should support!

  • PostgreSQL 10 Now Available in Beta on Heroku Postgres (Heroku)
    17 Oct 2017 15:49

    Earlier this month, PostgreSQL 10.0 was released. Today, we are excited to announce PostgreSQL 10 is available in beta on Heroku, bringing a number of notable feature and performance improvements to our managed PostgreSQL database service.

    The beta provides customers who want to try out the new release an easy way to do so, while customers who are happy with the current version can continue to stay on version 9.6 until we make version 10 generally available. Also, new databases will continue to default to version 9.6 until we release version 10 to GA.

    While PostgreSQL 10 has many feature and performance benefits, we’d like to highlight several that we are most looking forward to:

    Performance Improvements

    Most Heroku Postgres users will notice a performance boost when using PostgreSQL 10 for certain types of queries. The introduction of improved parallel queries, which optimize common types of JOINs and table scans, will result in faster queries on many kinds of datasets.

    Additionally, the introduction of multivariate statistics objects enables users to significantly enhance query performance on any datasets which contain correlated data: STATISTICS objects can be used to let the query planner know about relationships within your data. This section of the PostgreSQL 10 manual explains the feature in more detail.

    Native Table Partitioning

    You may know we support table partitioning on previous PostgreSQL versions through the pg_partman extension which allows both time-based and serial-based table partition sets. By automatically segmenting data into seperate physical stores, Postgres table partitioning can be a great way to keep good query performance for very large tables. In PostgreSQL 10, users will have the option to leverage native table partitioning.

    PostgreSQL 10 native partitioning is possible by column but also by any arbitrary expression: in the following example we partition the users table by the first letter of a user's name.

    CREATE TABLE users (
        id             serial not null,
        name           text not null,
        created_at     timestamptz not null
    PARTITION BY RANGE ( LOWER( LEFT( name, 1 ) ) );
    CREATE TABLE users_0
        partition of users (id, primary key (id), unique (name))
        for values from ('a') to ('f');
    CREATE TABLE users_1
        partition of users (id, primary key (id), unique (name))
        for values from ('f') to ('z'); 
    INSERT INTO users (name, created_at)
        VALUES ('camille', now()),
               ('greg', now());
    SELECT * FROM users_0;
    id |  name   |          created_at           
     1 | camille | 2017-10-13 18:45:17.882299+00
    (1 row)
    SELECT * FROM users_1;
     id | name |          created_at           
      2 | greg | 2017-10-13 18:45:17.882299+00
    (1 row)

    You can find more information on PostgreSQL native partitioning and its limitations on the PostgreSQL wiki.

    Other improvements and features we are excited about include replicated hash indexes and transaction status checking.

    Getting Started and Caveats

    You can add a Postgres 10 instance via the CLI:

    heroku addons:create heroku-postgresql --version 10
    heroku pg:info DATABASE_URL
    Plan:                  Hobby-dev
    Status:                Available
    PG Version:            10.0

    At present, all Heroku Postgres extensions - except for pg_partman, plv8 and redis_fdw - are supported while PostgreSQL 10 is in beta on Heroku. pg:backups is also supported on PostgreSQL 10.

    Additionally, pg:upgrade to PostgreSQL 10 is temporarily unavailable pending extension support and testing by our team. If you want to transfer data from your existing database, you can use pg:copy for now.

    Along with version 10 in beta, we currently support PostgreSQL versions 9.3, 9.4, 9.5 and 9.6 in GA. Once we gain confidence in this release through user testing and validation, we will make PostgreSQL 10 the default for all new databases on Heroku.

    Please contact us with any feedback, bugs, or questions at

  • Container Registry & Runtime GA: Deploy Docker Images to Heroku (Heroku)
    03 Oct 2017 16:00

    In the last few years Docker has emerged as a de facto standard for packaging apps for deployment. Today, Heroku Container Registry and Runtime is generally available, allowing you to deploy your Docker images directly to Heroku.

    With Container Registry, you get all of the benefits of Docker -- a great local development experience and flexibility to create your own stack -- with the benefits of running on Heroku: maintained infrastructure, container orchestration, routing, the leading add-ons ecosystem, and a world-class security & operations team.

    To deploy your Docker image to Heroku, simply run one command in the directory of your Dockerfile:

    $ heroku container:push web
    === Building web
    Step 1 : FROM alpine:latest
    Successfully built 74bab4bf0df3
    === Pushing web
    The push refers to a repository []
    c8821d626157: Pushed

    Heroku Container Registry allows you to easily build your Docker image locally and deploy to Heroku. Both Common Runtime and Private Spaces are supported.

    To get started, check out the Container Registry documentation.

    Use Docker to Develop Your Heroku App

    It can be especially frustrating when code that runs perfectly on your machine fails to run on another developer’s machine or a test/production environment. To solve this problem, many development teams are adopting container technologies, like Docker, to fully isolate their app from the underlying OS and local environment.

    Docker is well-suited for local development, because a container that runs on your machine gives you confidence that it will also work on another developer’s machine. As a Heroku developer you can now add Docker to your toolbelt and take advantage of the local development experience it provides. When your code is ready, simply push the image directly to Heroku.

    Create Your Own Stack

    Traditionally, applications deployed on Heroku use a Ubuntu-based stack. With Container Registry you can use any base image, such as Alpine. You can also install any dependency, giving you choice in the stack that you run on Heroku.

    In this example Dockerfile, we use an Alpine image, install the Python runtime, as well as ImagicMagick from the apk package manager.

    # Grab the latest alpine image
    FROM alpine:latest
    # Install python runtime
    RUN apk --update add python3 py-pip py-gunicorn py-psycopg2 bash 
    # Install ImageMagick
    RUN apk add imagemagick
    # Install dependencies
    ADD ./python-getting-started/requirements.txt /tmp/requirements.txt
    RUN pip install -qr /tmp/requirements.txt
    # Add our code
    ADD ./python-getting-started  /opt/webapp/
    WORKDIR /opt/webapp
    # Run the app.  CMD is required to run on Heroku            
    CMD gunicorn gettingstarted.wsgi --log-file -

    How Is Container Registry Different than git push heroku master?

    When you run git push heroku master your app is built with a buildpack and runs on a Ubuntu-based stack, both of which are maintained by Heroku. We provide a curated experience ensuring the OS, binaries, language runtime, and popular dependencies are up-to-date.

    By using Docker you get the flexibility of defining your own stack, plus the benefits of running on Heroku, such as maintained infrastructure, a security and SRE team, platform orchestration of containers & routing, and a robust ecosystem of add-on services. It’s your stack to curate, backed by Heroku.

    We’re Just Getting Started

    Today, Container Registry and Runtime provides you with more choice; choice in tools you use to develop and choice in stack you use in your app. And with Container Registry and Runtime, we’re just getting started; exciting new features for customizing your build are on the horizon. If you have feedback about Container Registry, we’d love to hear from you:

  • In the Cloud, No One Can Hear Your OutOfMemoryError (Heroku)
    02 Oct 2017 15:24

    Pushing an app to the cloud can feel like launching a probe into space. Once your project is thousands of miles away you can't bang on it with a hammer or replace broken parts when there's a problem. Your debugging efforts must rely on the instrumentation, telemetry, and remote controls included with the app when it was deployed. On Heroku, we've gladly done some of that prep work for you.

    Two new Heroku features, Heroku Exec and Language Runtime Metrics, improve your production monitoring, inspecting, and debugging experience on the platform. With Heroku Exec, you can create secure TCP and SSH tunnels into a dyno, which facilitate SSH sessions, port forwarding, remote debugging, and most popular diagnostic tools. Our Language Runtime Metrics feature supplements these tools by displaying detailed time series metrics in your Heroku dashboard.

    You can try Heroku Exec right now by running the following command on any Heroku app:

    $ heroku ps:exec
    Establishing credentials... done
    Connecting to web.1 on ⬢ evening-lowlands-62983...
    ~ $

    This creates an SSH session into a running dyno, which makes it possible to inspect your application's Java process with commands like jstat and jcmd. To make this even easier, we've integrated the most popular tools, including jmap and VisualVM, directly into the Heroku CLI so you can run commands such as heroku java:jmap to get a heap dump from a dyno.

    Let's take a closer look at how you can use these features to debug real production problems by testing them on a Java application with a memory leak.

    Inspecting an App with the Heroku Java CLI

    You can follow the steps in this section with your own Java app running on Heroku, or you can use the sample app we've created, which includes a synthetic memory leak. You'll use the Heroku Java CLI to inspect the process with VisualVM and other diagnostic tools to identify the source of the leak.

    Make sure you have a Heroku account and have installed the Heroku CLI, then run the following commands to deploy the sample app:

    $ git clone
    $ cd java-memory-leak
    $ heroku create
    $ git push heroku master

    Open the app in a browser by running:

    $ heroku open

    The page will tell you exactly how much memory has been leaked, and it will grow each time you refresh. Unfortunately, most real-world leaks are not this polite, and must be uncovered with monitoring tools. The Java Development Kit (JDK) comes bundled with several of these tools, a few of which are integrated into the Heroku Java CLI. To start, install the Java CLI plugin by running this command:

    $ heroku plugins:install heroku-cli-java

    Now use the CLI to connect a local VisualVM session, a lightweight profiler for the Java virtual machine, to your remote Heroku dyno by running:

    $ heroku java:visualvm

    The command will create the connection and open VisualVM. You can see a demonstration in this video:

    Refresh your app's web page a few times. As the memory leak grows, you'll also see the process's heap memory grow in VisualVM. This helps you identify that the leak is confined to heap memory as opposed to off-heap memory. But it doesn't narrow down the objects at the source of the growth. For that, you can use the jmap command like this:

    $ heroku java:jmap --hprof
    Establishing credentials... done
    Generating heap dump for web.1 on ⬢ pure-tor-76648...
    Dumping heap to /app/heapdump-5865f5e1-956a-4f60-8d66-8571b645a950.hprof ...
    Heap dump file created
    Downloading... ████████████████████████▏ 100% 00:00

    This downloads a binary heap dump from the dyno, which can then be imported into tools like Eclipse Memory Analyzer (MAT) for analysis. Indeed, when you open the HPROF file in Eclipse MAT, its leak suspects report correctly identifies that the Servlet is holding on to some very large FakeLeak objects:

    Eclipse Mat

    Eclipse MAT, VisualVM, and jmap are great when you know you have a problem, but they are not tools you'll want to run continuously. For passive metrics monitoring, you need a low-overhead, zero-fuss solution.

    Monitoring a Production App with Heroku JVM Metrics

    The Heroku Language Runtime Metrics feature gathers detailed performance information from your application process and displays it in the Heroku Dashboard. This differs from our existing application metrics, which are gathered from the operating system and the Heroku platform. An application process's runtime can report more specific metrics, such as heap memory use and garbage collection activity, which provide better visibility into how your code is working.

    To enable this feature, you'll first need to upgrade to a paid, Standard or above, dyno. Then add the Heroku Metrics buildpack to your app by running the following commands:

    $ heroku buildpacks:add -i heroku/metrics

    Finally, make an empty commit to your application, and redeploy:

    $ git commit --allow-empty -m "Add Heroku Metrics Buildpack"
    $ git push heroku master

    Browse to the Heroku Dashboard for your application, and view the Metrics tab. Below the standard memory and dyno load charts, you'll see the JVM metrics:


    These charts display heap memory (memory used for each object your program creates), non-heap memory (which includes Metaspace and buffer pools), and garbage collection activity. The metrics are reported by a small Java agent that Heroku attaches to your application process. This agent uses Prometheus, an open-source systems monitoring and alerting toolkit. We've chosen Prometheus because it's mature, has an active community, and is independent open-source project governed by the Cloud Native Computing Foundation.

    Heroku Language Metrics support for JVM languages is generally available, and support for Go is in beta. We'll have more languages coming soon.

    Bring Your Java Apps Down to Earth

    Your app may be physically running on servers in another country or continent, but that doesn't mean you have to forgo the tooling that helps you solve problems. We're lucky to have such excellent monitoring and debugging tools in the Java ecosystem, and at Heroku we're happy to make them easily available for you.

    For more information on the Heroku Java CLI, see the Heroku Exec article in Dev Center. For more information on JVM Runtime Metrics, see the Language Runtime Metrics article in Dev Center.

  • Heroku Exec and Language Runtime Metrics GA: Runtime Debugging on Heroku (Heroku)
    28 Sep 2017 15:54

    We’ve all been there -- you push your code to production and a leak causes memory usage to grow out of control. To determine the root cause of the problem, you need to be able to monitor, inspect, and debug the production application, collecting detailed data at runtime.

    Today we’re making it even easier to debug your applications on Heroku, with the general availability of Language Runtime Metrics, starting with JVM languages, and Heroku Exec. Language metrics surfaces key indicators of an issue, like garbage collection activity, and heap and non-heap memory usage, on a unified timeline in the Heroku Dashboard.

    After you’ve identified an issue, you can use Exec to connect to a dyno at runtime, via SSH, for further inspection and remote debugging. When combined with Application Metrics, Language Runtime Metrics and Exec provide a robust set of tools for maintaining production application health.

    Runtime Inspection with Exec: SSH to the Dyno

    When a problem arises, and you need to collect data at runtime, you can connect to a dyno via SSH with Exec:

    $ heroku ps:exec
    Establishing credentials... done
    Connecting to web.1 on ⬢ ns-pipeline-staging... 
    ~ $ top

    Unlike heroku run bash, which creates a one-off dyno, Exec makes an SSH connection directly to an existing dyno (e.g., web.2). Exec also allows you to copy files off of a dyno, forward traffic on a local port to a dyno, and take advantage of common Java debugging tools.

    Connecting a Remote Debugger & Java Tools

    While top and ps are great for debugging memory issues, sometimes you need to connect a remote debugger, such as Eclipse, IntelliJ or WebStorm, and step through code. With Exec, you can forward traffic from a local port to a dyno, enabling you to connect your remote debugger.

    $ heroku ps:forward 5858

    To see an example, check out the video above.

    For Java applications, you can use JVM-specific commands to emit jmap and jstack dumps. Exec also offers the ability to connect JConsole and VisualVM to an application with one command.

    Monitoring Production Applications with Language Runtime Metrics

    Language metrics are displayed within the Application Metrics dashboard so you can easily correlate language runtime health to overall application health.


    Language runtime metrics uses the prometheus agent, which provides a lightweight method for gathering data. The additional metrics collection has minimal impact on application performance. The initial release supports JVM languages, with additional languages in progress.

    We’d Love Your Feedback

    Exec and Language Runtime Metrics are now generally available and we have plans to extend our language-specific metrics and tools support beyond Java. If you have feedback on either product, or want to share tools you would like us to support in the future, please let us know!

  • Announcing the Dublin, Ireland Region for Heroku Private Spaces (Heroku)
    26 Sep 2017 15:51

    We are excited to announce the Dublin region for Heroku Private Spaces is now generally available for Heroku Enterprise customers. Dublin joins the growing list of regions that Private Spaces supports: Sydney, Virginia, Oregon, Frankfurt, and Tokyo. With the Private Spaces Dublin region, organizations can build and deploy Heroku-style apps closer to their UK customers, reducing network latency and providing a better user experience.

    Heroku Private Spaces, available as part of Heroku Enterprise, is a network isolated group of apps and data services with a dedicated runtime environment, provisioned by Heroku in a geographic region you specify. With Spaces you can build modern apps with the powerful Heroku developer experience and get enterprise-grade secure network topologies. This enables your Heroku applications to securely connect to on-premise systems on your corporate network and other cloud services, including Salesforce.


    To create a Private Space in Dublin, select the Spaces tab in Heroku Dashboard in Heroku Enterprise, then click the “New Space” button and choose “Dublin, Ireland” from the the Space Region dropdown.

    Or you can create the space in the CLI by running:

    $ heroku spaces:create my-new-space --region dublin --team my-team
    Creating space my-new-space in team my-team... done

    After a Private Space in Dublin is created, Heroku apps can be created inside it as normal. All of Heroku’s data add-ons, Postgres, Redis, and Kafka, are also available in Dublin as are a variety of third-party Add-ons.


    All Heroku Enterprise customers can immediately begin to create Private Spaces in Dublin and deploy apps there. If there are regions which you would like us to support in the future, please let us know.

  • Kafka Everywhere: New Plans and Pricing for Apache Kafka on Heroku (Heroku)
    14 Sep 2017 15:32

    Event-driven architectures are on the rise, in response to fast-moving data and constellations of inter-connected systems. In order to support this trend, last year we released Apache Kafka on Heroku - a gracefully integrated, fully managed, and carefully optimized element of Heroku's platform that is the culmination of years of experience of running many hundreds of Kafka clusters in production and contributing code to the Kafka ecosystem.

    Today, we are excited to announce additional plans and pricing in our Kafka offering in order to make Apache Kafka more accessible, and to better support development, testing, and low volume production needs.

    Apache Kafka on Heroku: Now With More Flexibility and Speed

    Apache Kafka is a powerful, distributed streaming platform, and the dominant open source solution in managing high scale event streams. Kafka enables you to easily design and implement architectures for many important use cases, such as elastic queuing, data pipelines & analytics, and microservices coordination. Apache Kafka on Heroku removes the complexity and cost of running Kafka, making its valuable resources available to a broad range of developers and applications.

    The new addition to our managed Kafka plans, Basic, is based on a robust multi-tenant Kafka architecture. Multi-tenancy provides for much faster access to Kafka, with new resources being available in a matter of seconds, rather than minutes, and a much more accessible price point. This allows us to better serve application creators needing Kafka configurations that are more suitable for development or staging, or as production environments that don't require the full capacity of a dedicated cluster (as provided in our existing Standard and Extended plans).

    Basic Standard Extended
    Cluster Shared Dedicated Dedicated
    Event Stream Volume Low High Massive
    Ideal Use Case Develop + Test,
    Low-Scale Production
    Production Production
    Price ($/month) $100+ $1,000+ $4,000+

    These new multi-tenant plans allow for rapid creation of Kafka add-ons, easy integration into Heroku applications, and the world-class monitoring and tooling you have come to expect from Heroku.

    Get Started Today

    Using one of the new plans is as simple as provisioning an instance of the Kafka add-on and attaching it to a Heroku app:

    heroku addons:create heroku-kafka:basic-0 -a sushi-app

    We are excited to see what you build with Kafka! Full details of the new Kafka plans can be found in Heroku Elements and in Heroku Dev Center. If you have any questions or feedback, please let us know at

  • Best of the Blogs: A Heroku Community Tour (Heroku)
    13 Sep 2017 15:36

    Heroku is very fortunate to have a strong community of developers that are excited and passionate about our product. Every day we hear from customers who tell us how much easier Heroku has made their lives, and they frequently share stories about interesting technical projects we've helped them bring to life.

    Our customers love us, and we love them right back. Today we'll take a look at a few blog posts and applications from Heroku users that illustrate what makes our community so special. We hope you enjoy the tour. If you have Heroku stories of your own you'd like to share, we'd love to hear them!

    Dynos Spinning Other Dynos with Heroku

    This article comes to us from Yoni Weisbrod, a JS & React Native developer from Ivy. Ivy makes a community and business management tool for interior designers. They ran into some challenges when their worker dynos started maxing out their memory while generating some particularly hefty internal reports.

    Yoni and team decided to create a couple of new workers named Julia and Winston (after the characters from 1984). They use their new workers to dynamically handle the increased load from their report generation tasks. Julia uses the Heroku API to spin up Winston whenever a new report needs to be generated, then adds the report job to Winston's queue. Winston generates the report on demand and spins back down when the mischief has been managed.

    The beauty of this solution is that Winston is only used for the exact period of time it takes to generate the report, and since Heroku prorates dyno usage to the second, Ivy only ends up paying for the time they need to get their reports.

    How to Create and Deploy a Telegram Bot

    Imagine you're chatting with a friend on Telegram and you're reminded of a hilarious cat video you want to share. Think of all the tedium you'll have to endure while you switch applications, search for the video, get a shareable link, switch back to Telegram, and finally paste the link. This is the 21st century! We shouldn't have to find our own videos. This sounds like the perfect use case for a bot.

    You can read all about building your own basic greeting bot in this article from Roman Gaponov, CEO and Co-Founder of the DjangoStars agile software development company from Kiev, Ukraine. Roman uses Python on Heroku with the Telegram Bots API to build a simple bot that replies with a distinct message based on the time of day.

    If you're interested in building the YouTube lookup bot described above, you'll be much closer after you've walked through the development process described in Roman's article. Have a look at the YouTube Data API to get the rest of the way there.

    Werewolf Helper App

    Gabriela Gonzalez, a software engineer at Kibo Commerce, is a fantastic digital artist and the creator of the Werewolf Helper app on Heroku.

    The Werwolf Game (also known as Mafia) was originally created by Dmitry Davidoff. It's a popular game among developer communities because it requires minimal setup and offers a unique experience every time you play it.

    Gabriela's beautiful site brings Werewolf to your online friends as well, with a shared deck online that allows you and your frenemies to play remotely over Skype or any other voice medium. Now you have no excuses not to bond remotely over terror under a full moon.

    Painless PostgreSQL + Django

    Eleanor Stribling brings us a well-written post detailing PostgreSQL setup for Django applications. When you first set up a new Django app, it uses SQLite by default. Because Heroku uses an ephemeral file system (and SQLite is a disk-backed database), it's very likely that you would lose data if you did manage to get SQLite to work on Heroku.

    Heroku is pretty good at warning you when we detect that you're attempting to use SQLite, so you're not likely to actually lose any data. If you'd like some more information about SQLite on Heroku, check out this devcenter article.

  • Heroku Webhooks: Powering New Integrations and Real-time Notifications (Heroku)
    22 Aug 2017 15:44

    We're happy to announce that Heroku app webhooks is now generally available for all Heroku customers.

    App webhooks provide notifications when your Heroku app changes, including modifications to domain settings, releases, add-ons, and dyno formations. These notifications can empower your internal communications, dashboards, bots or anything else that can receive HTTP POST requests. Integrating with Heroku webhooks provides easy support for driving custom workflows and 3rd party tools.

    Creating webhooks

    With the webhooks CLI plugin, you can subscribe to events with a single command.

    heroku plugins:install heroku-webhooks        
    heroku webhooks:add -i api:release -l notify -u -a your-app  

    In this example, after a new release is created for your-app, Heroku sends a POST request to your server endpoint with the details of the event. The example below shows the first section of a typical POST request:


    Receiving webhooks

    Webhooks are delivered by POST requests to a designated, publicly accessible URL. By specifying the sync notification level when setting up webhooks, Heroku will automatically retry delivery of failed webhook events.

    When to Use webhooks

    Integrating with webhooks simplifies reacting to changes from your Heroku apps. Instead of repeatedly polling the Platform API and comparing the responses to identify changes, a webhook will automatically be delivered when a change has occurred.

    Webhooks can power many real-time use cases, from internal dashboards that aggregate events from your Heroku applications to customized email notifications for specific members of your team. Chat notifications are a great example of how webhooks can be used to keep your team informed. While Heroku ChatOps provides a ready-built solution for pipeline events in Slack, you can also use webhooks to build custom notifications.

    Easy to Setup

    The app webhooks tutorial allows you to deploy a “webhook viewer” application in just a few clicks. The application walks you through creating a webhook and allows you to check the webhook contents before integrating webhook reception into your own application. The application also enables easier debugging as you’re able to easily view all the HTTP request data.

    After completing the tutorial, learn about all the supported event types and the webhooks API in the full webhooks documentation.


    We plan to support more event types in the future and welcome your suggestions. If you have feedback you can reach us at

  • Evolution of the Heroku CLI: 2008-2017 (Heroku)
    15 Aug 2017 15:45

    Over the past decade, millions of developers have interacted with the Heroku CLI. In those 10 years, the CLI has gone through many changes. We've changed languages several times; redesigned the plugin architecture; and improved test coverage and the test framework. What follows is the story of our team's journey to build and maintain the Heroku CLI from the early days of Heroku to today.

    1. Ruby (CLI v1-v3)
    2. Go/Node (CLI v4)
    3. Go/Node (CLI v5)
    4. Pure Node (CLI v6)
    5. What's Next?

    Ruby (CLI v1-v3)

    Our original CLI (v1-v3) was written in Ruby and served us well for many years. Ruby is a great, expressive language for building CLIs, however, we started experiencing enough problems that we knew it was time to start thinking about some major changes for the next version.

    For example, the v3 CLI performed at about half the speed on Windows as it did on Unix. It was also difficult to keep the Ruby environment for a user's application separate from the one used by the CLI. A user may be working on a legacy Ruby 1.8.7 application with gems specific to Ruby 1.8.7. These must not conflict with the Ruby version and gem versions the CLI uses. For this reason, commands like heroku local (which came later) would have been hard to implement.

    However, we liked the plugin framework of the v3 CLI. Plugins provide a way for us to nurse new features, test them first internally and then later in private and public beta. Not only does this allow us to write experimental code that we don't have to ship to all users, but also, since the CLI is an open-source project, we sometimes don't want to expose products we're just getting started on (or that are experimental) in a public repository. A new CLI not only needed to provide a plugin framework like v3, but also it was something we wanted to expand on as well.

    Another reason we needed to rewrite the CLI was to move to Heroku's API v3. At the start of this project, we knew that the old API would be deprecated within a few years, so we wanted to kill two birds with one stone by moving to the new API as we rewrote the CLI.

    Go/Node (CLI v4)

    When we started planning for v4, we originally wanted the entire CLI to be written in Go. An experimental CLI was even done before I started at the company to rebuild the CLI in Go called hk. hk was a major departure from the existing CLI that not only changed all the internals, but changed all the commands and IO as well.

    Parity with CLI v3

    We couldn't realistically see a major switch to a new CLI that didn't keep at least a very similar command syntax. CLIs are not like web interfaces, and we learned this the hard way. On the web you can move a button around, and users won't have much trouble seeing where it went. Renaming a CLI command is a different matter. This was incredibly disruptive to users. We never want users to go through frustration like that again. Continuing to use existing syntax and output was a major goal of this project and all future changes to the CLI.

    While we were changing things, we identified some commands that we felt needed work with their input or output. For example, the output of heroku addons changed significantly using a new table output. We were careful to display deprecation warnings on significant changes, though. This is when we first started using color heavily in the CLI. We disable color when the output is not a tty to avoid any issues with parsing the CLI output. We also added a --json option to many commands to make it easier to script the CLI with jq.

    No Runtime Dependency

    In v3, ensuring that we had a Ruby binary that didn't conflict with anything on the user's machine on all platforms was a big headache. The way it was done before also did not allow us to update Ruby without installing a new CLI (so we would've been stuck with Ruby 1.9 forever). We wanted to ensure that the new CLI didn't have a runtime dependency so that we could write code in whatever version of Ruby we wanted to without worrying about compatibility.

    So Why Not All Go?

    You might still be wondering why we didn’t reimplement both the plugins and core in Go (but maintain the same command syntax) to obviate our runtime dependency concerns. As I mentioned, originally we did want to write the CLI in Go as it provided extremely fast single-file binaries with no runtime dependency. However, we had trouble reconciling this with the goal of the plugin interface. At the time, Go provided no support for dynamic libraries and even today this capability is extremely limited. We considered an approach where plugins would be a set of compiled binaries that could be written in any language, but this didn't provide a strong interface to the CLI. It also begged the question of where they would get compiled for all the architectures.

    Node.js for Plugins and Improved Plugin Architecture

    This was when we started to think about Node as the implementation language for plugins. The goal was for the core CLI (written in Go) to download Node just to run plugins and to keep this Node separate from any Node binary on the machine. This kept the runtime dependency to a minimum.

    Additionally, we wanted plugins to be able to have their own dependencies (library not runtime). Ruby made this hard as it's very difficult to have multiple versions of the same gem installed. If we ever wanted to update a gem in v3, we had to go out of our way to fix every plugin in the ecosystem to work with the new version. This made updating any dependencies difficult. It also didn't allow plugins to specify their own dependencies. For example, the heroku-redis plugin needs a redis dependency that the rest of the CLI doesn't need.

    We also wanted to improve the plugin integration process. In v3, when we wanted the functionality from a plugin to go into the core of the CLI, it was a manual step that involved moving the commands and code into the core of the CLI and then deprecating the old plugin. It was fraught with errors and we often had issues come up attempting this. Issues were compounded because it usually wasn't done by a CLI engineer. It was done by a member on another team that was usually moving a plugin for their first time.

    Ultimately we decided to flip this approach on its head. Rather than figure out an easy way to migrate plugin commands into the core, we made the CLI a collection of core plugins. In other words, a plugin could be developed on its own and installed as a “user plugin”, then when we wanted to deliver it to all users and have it come preinstalled, we simply declared it as a “core plugin”. No modifications to the plugin itself would be required.

    This model provided another benefit. The CLI is now a modular set of plugins where each plugin could potentially be maintained by a separate team. The CLI provides an interface that plugins must meet, but outside of that, individual teams can build their own plugins the way they want without impacting the rest of the codebase.

    Allowing these kinds of differences in plugins is actually really powerful. It has allowed developers on other teams and other companies to provide us with clever ideas about how to build plugins. We've continually been able to make improvements to the plugin syntax and conventions by allowing other developers the ability to write things differently as long as they implemented the interface.

    Slow Migration

    One thing I've learned from doing similar migrations on back-end web services is that it's always easier to migrate something bit-by-bit rather than doing a full-scale replacement. The CLI is a huge project with lots of moving parts. Doing a full-scale replacement would have been a 1+ year project and would have involved a painful QA process while we validated the new CLI.

    Instead, we decided to migrate each command individually. We started out by writing a small core CLI with just a few lesser-used commands and migrating them from the v3 CLI to the v4 CLI one at a time. Moving slow allowed us to identify issues with specific commands (whether it was an issue with the core of the CLI, the command itself, or using the new API). This minimized effort on our part and user impact by allowing us to quickly jump on issues related to command conversion.

    We knew this project would likely take 2 years or longer when we started. This wasn't our only task during this time though, so it enabled us to make continual progress while also working on other things. Over the course of the project, we sometimes spent more time on command conversion, sometimes less. Whatever made sense for us at the time.

    The only real drawback with this approach was user confusion. Seeing two versions of the CLI listed when running heroku version was odd and it also wasn't clear where the code lived for the CLI.

    We enabled the gradual migration from v3 to v4 by first having v3 download v4, if it did not exist, into a dotfile of the user's home directory. v4 provides a hidden command heroku commands --json that outputs all the information about every command including the help. When v3 starts, it runs this command so that it knows what commands it needs to proxy to v4 as well as what the full combined help is for both v3 and v4.

    For 2 years we shipped our v4 Go/Node CLI alongside v3. We converted commands one by one until everything was converted.

    Go/Node (CLI v5)

    The v5 release of the CLI was more of an incremental change. Users would occasionally see issues with v4 when first running the CLI because it had trouble downloading Node or the core plugins. v5 was a change from downloading Node when the CLI was first run, to including Node in the initial tarball so it would be available when the CLI first loaded. Another change was that instead of running npm install to install the core plugins on first run, we included all the core plugins' Node files with the initial tarball and kept the user plugins separate.

    Ruby to Node Command Conversion Complete

    In December 2016, we finally finished converting all the commands into the new plugins-based CLI. At this point we modified our installers to no longer include the v3 CLI and the shim that launched the v4 or v5 CLI. Existing users with the CLI already installed as of this time will still be using the v3 CLI because we can't auto-update all parts of the CLI, but new installers will not include v3 and are fully migrated to v5 (or now, v6). If you still have the Ruby CLI installed (you’ll know if you run ‘heroku version’ and see v3.x.x mentioned), you’ll benefit from a slight speed improvement by installing the current version of the CLI to get rid of these old v3 components.

    Pure Node (CLI v6)

    In April 2017 we released the next big iteration of the CLI, v6. This included a number of advantages with a lighter and generic core written only in Node that could be used as a template for building other CLIs, and a new command syntax.

    Leaving Go

    While at Heroku we use Go heavily on the server-side with great success, Go did not work out well for us as a CLI language due to a number of issues. OS updates would cause networking issues and cross-compiling would cause issues where linking to native objects did not work. Go is also a relatively low-level language which increased the time to write new functionality. We were also writing very similar, if not exactly the same, code in Ruby and Node so we could directly compare how difficult it was to write the same functionality in multiple languages.

    We had long felt that the CLI should be written in pure Node. In addition to only having one language used and fewer of the issues we had writing the CLI in Go, it also would allow for more communication between plugins and the core. In v4 and v5, the CLI started a new Node process every time it wanted to request something from a plugin or command (which takes a few hundred ms). Writing the CLI entirely in Node would keep everything loaded in a single process. Among other things, this allowed us to design a dynamic autocomplete feature we had long wanted.


    Occasionally we would be asked how other people could take advantage of the CLI codebase for their own use — not just to extend the Heroku CLI, but to write entirely new CLIs themselves. Unfortunately the Node/Go CLI was complicated for a few reasons: it had a complex Makefile to build both languages and the plugins, it was designed to work both standalone as well as inside v3, and there was quite a bit of “special” functionality that only worked with Heroku commands. (A good example is the --app flag). We wanted a general solution to allow other potential CLI writers to be able to have custom functionality like this as well.

    CLI v6 is built on a platform we call cli-engine. It's not something that is quite ready for public use just yet, but the code is open sourced if you'd like to take a peek and see how it works. Expect to hear more about this soon when we launch examples and documentation around its use.

    New Plugin Interface

    Due to the changes needed to support much of the new functionality in CLI v6, we knew that we would have to significantly change the way plugins were written. Rather than look at this as a challenge, we considered it an opportunity to make improvements with new JavaScript syntax.

    The main change was moving from the old JavaScript object commands into ES2015 (ES6) class-based commands.

    // v5
    const cli = require('heroku-cli-util')
    const co = require('co')
    function * run (context, heroku) {
      let user = context.flags.user || 'world'
      cli.log(`hello ${user}`)
    module.exports = {
      topic: 'hello',
      command: 'world',
      flags: [
        { name: 'user', hasValue: true, description: 'who to say hello to' }
      run: co.wrap(cli.command(run))
    // v6
    import {Command, flags} from 'cli-engine-heroku'
    export default class HelloCommand extends Command {
        static topic = 'hello'
        static command = 'world'
        static flags = {
          user: flags.string({ description: 'who to say hello to' })
        async run () {
          let user = this.flags.user || 'world'
          this.out.log(`hello ${user}`)


    async/await finally landed in Node 7 while we were building CLI v6. We had been anticipating this since we began the project by using co. Switching to async/await is largely a drop-in replacement:

    // co
    const co = require('co')
    let run = co.wrap(function * () {
      let apps = yield heroku.get('/apps')
    // async/await
    async function {
      let apps = await heroku.get('/apps')

    The only downside of moving away from co is that it offered some parallelization tricks using arrays of promises or objects of promises. We have to fall back to using Promise.all() now:

    // co
    let run = co.wrap(function * () {
      let apps = yield {
        a: heroku.get('/apps/appa'),
        b: heroku.get('/apps/appb')
    // async/await
    async function run () {
      let apps = await Promise.all([

    It's not a major drawback, but it does make the code slightly more complicated. Not having to use a dependency and the semantic benefits of using async/await far outweigh this drawback.


    The CLI is now written with Flow. This static type checker makes plugin development much easier as it can enable text editors to provide powerful code autocomplete and syntax checking, verifying that the plugin interface is used correctly. It makes plugins more resilient to change by providing interfaces checked with static analysis.

    2017-07-20 09

    While learning new tools is a challenge when writing code, we've found that with Flow the difficulty was all in writing the core of the CLI and not as much in plugin writing. Writing plugins involves using existing types and functions so often plugins won't have any type definitions at all, where the core has many. This means we as the CLI engineers have done the hard work to include the static analysis, but plugin developers reap the benefits of having their code checked without having to learn much of a new tool if any.


    Class properties and Flow required us to use Babel in order to preprocess the code. Because the process for developing plugins requires you to “link” plugins into the CLI, this allowed us to check if the code had any changes before running the plugin. This means that we can use Babel without requiring a “watch” process to build the code. It happens automatically and there is no need to setup Babel or anything else. All you need is the CLI to develop plugins. (Note that Node must be installed for testing plugins, but it isn't needed to run a plugin in dev mode.)

    Improved Testing

    Testing is crucial to a large, heavily-used CLI. Making changes in the core of the CLI can have unexpected impact so providing good test coverage and making tests easy to write well is very important. We've seen what common patterns are useful in writing tests and iterated on them to make them concise and simple.

    As part of the new plugin interface, we've also done some work to make testing better. There were some gaps in our coverage before where we would have common issues. We worked hard to fill those gaps, ensuring our tests guaranteed commands were properly implementing the plugin interface while keeping the tests as simple as possible to write. Here is what they look like in comparison from v5 of the CLI to v6:

    // v5 mocha test: ./test/commands/hello.js
    const cli = require('heroku-cli-util')
    const expect = require('chai').expect
    describe('hello:world', function () {
      beforeEach(() => {
      it('says hello to a user', function () {
        return{flags: {user: 'jeff'}})
          .then(() => expect(cli.stdout).to.equal('hello jeff!\n'))
    // v6 jest test: ./src/commands/hello.test.js
    import Hello from './hello'
    describe('hello:world', () => {
      it('says hello to a user', async () => {
        let {stdout} = await Hello.mock('--user', 'jeff')
        expect(stdout).toEqual('hello jeff!\n')

    The syntax is almost identical, but we're using Jest in v6 and Mocha in v5. Jest comes preloaded with a mocking framework and expectation framework so there is much less to configure than with mocha.

    The v6 tests also run the flag parser which is why '--user', 'jeff' has to be passed in. This avoids a common issue with writing v5 tests where you could write a test that works but not include the flag on the command definition. Also, if there is any quirk or change with the parser, we'll be able to catch it in the test since it's running the same parser.

    What's Next?

    With these changes in place, we've built a foundation for the CLI that's already been successful for several projects at Heroku. It empowers teams to quickly build new functionality that is well tested, easy to maintain, and has solid test coverage. In addition, with our CLI Style Guide and common UI components, we're able to deliver a consistent interface.

    In the near future, expect to see more work done to build more interactive interfaces that take advantage of what is possible in a CLI. We're also planning on helping others build similar CLIs both through releasing cli-engine as a general purpose CLI framework, but also through guidelines taken from our Style Guide that we feel all CLIs should strive to meet.

  • Heroku Postgres Update: Configuration, Credentials, and CI (Heroku)
    08 Aug 2017 15:13

    At the core of Heroku’s data services sits Postgres, and today, we are making it even easier to bend Heroku Postgres to the very unique needs of your application’s stack. With these new features, you can easily customize Postgres, making it more powerful and configurable, while retaining all the automation and management capabilities of Heroku Postgres you know and love. By changing Postgres settings, creating and working with database credentials, and providing tight integrations to Heroku and Heroku CI, you now have the ability to further tune your Postgres database to your team’s needs.

    More Flexible Postgres with PGSettings

    As we start peeling back the layers of Heroku Postgres, the ability to change the default behavior is the first step in making Heroku Postgres more flexible. Using the Heroku CLI, any developer can use the PGSettings feature to change portions of the default Heroku Postgres configuration. One of the more acute areas to change behavior is around database logging. As Heroku Postgres databases start getting sufficiently large, this could be in terms of the number of transactions, data volumes, or connections, these databases are generating large amount of logs that could hamper performance of the database.

    $ heroku pg:settings postgresql-large-1234 -a sushi
    === postgresql-large-1234
    log-lock-waits:             true
    log-min-duration-statement: 2000
    log-statement:              ddl

    With log-statements, for example, you can change this setting to none and the database won’t log any information besides slow queries and errors. For databases on Heroku Postgres that experience a large amount of thrashing in the schema, for example, temp tables that come and go, this can save lots of space and increase performance of the system by not having to write so many log statements.

    $ heroku pg:settings:log-statement ddl -a sushi
    log-statement has been set to ddl for postgresql-large-1234.
    All data definition statements, such as CREATE, ALTER and DROP, will be logged in your application's logs.

    PGSettings is available for Heroku Postgres databases running on a current production plan, Standard, Premium, Private, or Shield, on Postgres version 9.6 or above.

    Manage Access Permissions with Heroku Postgres Credentials

    One of the benefits of Heroku Postgres is how the relationship between a database and its associated application are automatically maintained. When a database needs to be re-instantiated, or its credentials have changed, these values are automatically reflected to the application.

    With the advent of sharable add-ons (and databases), a single database can now be used by multiple applications. With Heroku Postgres Credentials, each of those applications can now have a unique connection to each database, so the same automated credential management can now carry across multiple applications.

    In addition, as these credentials are scoped to a Heroku application, this feature provides an easy and powerful way to manage the scoping of database access to discrete groups of users in a way that’s tightly integrated to the existing Heroku permissions model. For example, if your team has a data scientist or analyst, a best practice would be to create a read-only credential and give that individual access via that scoped credential which they can retrieve by accessing the environment information for a specific application. This way, the risk to changing the database has been mitigated.

    Credentials can be created via the Heroku Data dashboard or the Heroku CLI, and are available for all non-legacy production plans, Standard, Premium, Private and Shield, on Postgres 9.6 and above.


    Tight Integration With Heroku CI and Pipelines

    You can now auto-provision Heroku Postgres instances for all your Heroku CI test runs and Heroku Pipelines Review apps with zero configuration. This allows Heroku Postgres users seamless, automated environment management for testing the code of any pull request, Git merge, or arbitrary build against a free, disposable Heroku Postgres hobby instance. You can optionally populate these instances with test data in the "Release Phase" script fully support by Heroku Pipelines. Each ephemeral Heroku Postgres DB is designed to instantiate (and self-destroy) quickly and cleanly -- and you pay only for the dynos during duration of the test run, or the existence of the Review app. As always, progress and results are reported to your development team in Slack, in GitHub, and your Heroku Dashboard.

    Integrating all the bits!

    All of these features - PGSettings, Heroku Postgres Credentials and Integration with Heroku CI and Pipelines - are available today. We are exploring new ways to make Postgres even more versatile and powerful, and if you have any new settings you’d like to see exposed, or any ideas on integrations with Heroku, email us at

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>