• Lead dev @BlueSolutions/Polyconseil: Autolib'
• Python/Django enthusiast:
  • factory_boy
  • xworkflows
• Contacts:
  • @raphaelbarrois
  • https://github.com/rbarrois
  • Xelnor@Freenode

• A set of autonomous, interconnected services
• Different codebase
• Each service with its own database

1. Why / when ?
2. Building it
3. RPCs
4. Keep it running

• In the beginning, a clean, neat design

• Two years later

• Two years later
  • New features

• Two years later
  • New features
  • Refactoring

• Two years later
  • New features
  • Refactoring
  • Auditing, logging

• Two years later
  • New features
  • Refactoring
  • Auditing, logging
  • Marketing & sales

• Two years later

New product/feature:

• Not an incremental improvement
• Strongly disconnected from the core one
• Need to launch fast

New installations of main product:

• Under another brand
• In another datacenter
• With a separate database (legal concerns)
• Operated by the same (ops) team

Technical improvements:

• Scaling / sharding
• Geographic expansion
• Debundling and upgrading

Extract generic code

• misc, util, tools
• Everything that isn't business-specific
• And some basic business components, too (common models, UI, etc.)

Put it into a separate modules => Adapt your deployment process for fast-moving internal dependencies

Use a single, coherent visual design:

• Extract your common UI templates
• Add inter-product headers
• Normalize your assets pipeline (bower, grunt)

Deploy an efficient Single-Sign-On solution:

• SSO, not "centralized authentication"
• Log-in once, for all services
• OpenID-like for web customers
• kerberos-based for internal users, if you can

• Efficient, secure authentication.
• Available with MS Active Directory, or open-source MIT implementation
• Works with Mac OS, Linux, Windows

Concept:

1. Authenticate against a secure authentication server ("KDC")
2. Receive a ticket
3. Connect to a server
4. Mutual ticket validation

Also works for websites

• Firefox, chrome, IE
• Apache, nginx (https://github.com/rbarrois/spnego-http-auth-nginx-module)
• Need to trust the site in configuration
• Fallback to ugly HTTP authentication box

=> Django receives a REMOTE_USER HTTP header

• Keep a single "source of truth"
• Split per business domain, not per object/table
• Think about redundancy / availability

Loosely coupled, isolated module:

• ACL management
• Registration
• Finance & billing

Avoid rewriting the whole module before splitting:

1. Enforce the target API within the source code, running on the same host
2. Start the remote module, running the exact same code
3. Add a dispatcher that calls both local and remote modules
4. Once data is properly feeding the new module, switch to "full remote" mode
5. Remove the dispatcher

At first, avoid conforming to a strict REST API:

• You know exactly which methods you need
• Avoids lots of performance issues
• Easier to port from internal function calls

Important points:

• Authentication & security
• Changing the schema
• Debugging and logging

Don't trust the frontend:

• Ask to push a "proof of identity"
• Check authorization against local ACLs

You'll go through several iterations of the API.

Avoid complex migrations:

1. Deploy client compatible with old and new contract
2. Deploy server with both old and new contract
3. Ask client to switch to new contract
4. Deploy client and server without old contract

Simpler:

1. Deploy new server, with new (backwards-compatible) contract
2. Deploy new client, aware of new features

Choose your protocol:

• Allows optional fields, with defaults
• Doesn't care about unknown fields
• Supports some path-like notion (/api/v2/resource/)

Three layers:

• Transport (HTTP, plain TCP, ZeroMQ, AMQP, ...)
• Serialization (JSON, msgpack, ...)
• RPC (JSON-RPC, SOAP, ...)

Required features:

• Load-balancing
• Message-oriented (with binary support)
• asyncio compatible
• Optional: Request/response and asynchronous

Options:

• HTTP: good at load-balancing, debugging; bad for binary data
• AMQP: good at load-balancing, async, binary; bad for time-critical request/response
• ZeroMQ: good at async, binary, perf; need to rollout your own load-balancing
• Plain TCP: rollout your own! - bad for message-oriented
• Plain UDP: rollout your own! - bad for request/response & monitoring

Required features:

• Complex objects
• Efficient packing
• Support for schema validation and updates
• Optional: enums, datetime, custom types

Options:

• msgpack: good for packing; no schema, no custom types
• protobuf: good for packing, schema; no custom types
• JSON: bad packing; no schema nor custom types
• XML: strict schema, advanced types; bad packing
• pickle: NO.

Required features:

• Authentication
• Explicit schema
• Unambiguous serialization

Provided by the implementation:

• Logging/debug
• Simple client calls, efficient server-side declaration
• Optional: asyncio compatible

Options:

• JSON-RPC: well-defined serialization; no auth, implicit schema ("any defined function")
• XML-RPC: well-defined serialization; no auth, implicit schema ("any defined function")
• SOAP/XML: auth, explicit schema; inconsistent serialization
• ZeroRPC: couldn't find a spec; no auth, implicit schema ("any defined function")
• JSON/REST: not a real RPC standard

Recommended technologies:

• Start with a simple stack:

  HTTP transport, JSON serialization, JSON-RPC/REST-like protocol

• Move to a more efficient stack for scaling/perf:

  ZeroMQ/AMQP transport, Protobuf serialization, <missing> protocol

• Reliability / redundancy
• Managing deploys and upgrades
• Splitting additional modules

Ensure that the role of each service is clear:

• Is it the only source for some data, and must be queried every time?

  => High availability required

• Is it an authoritative datastore, cached by client services?

  => Don't forget cache invalidation and automatic synchronization

New challenges:

• Inter-system dependencies
• Handling contract changes
• Shared libraries

• Critical service: progressive upgrade

  => Play with load-balancers

• Seldom used backends: up to you

• Client services: progressive upgrade

  => Ensure you're still compatible with your upstream services

• Small changes: deploy new server then new clients

  => Needs more steps for strict schemas

• Big changes: increase the API version number (new code path)

  => Monitor old service for non-upgraded clients

Upgrades to various libraries:

• Security

  => Deploy as soon as possible

• New major features

  => Integrate to a new major upgrade

• Minor improvements

  => For the next minor upgrade

Release often, if only to upgrade dependencies

• Avoids using obsolete code
• Helps with future feature-based upgrades
• Less versions of libraries in production => easier to track and fix bugs

Full range of tests:

• Current prod branch with pinned dependencies
• Current prod branch with latest version of dependencies
• Current master with pinned dependencies
• Current master with latest version of dependencies

Different levels:

• Server-side contracts

  => Proper input validation

• Isolated clients

  => Runs faster

• Full-system tests

  => Most accurate

• Connect a fake client with an in-memory transport
  • Tests the whole stack (including errors, logging)
  • Fast
• Send already deserialized objects to actual methods
  • Skips some layers
  • Slightly faster
  • Easier tests on returned values
• Connect an actual client with a real transport
  • Closer to real setups
  • Allows some failure modes (client disconnects in flight)

• Required when network is involved (until aiodjango? :P)
• Used by some production code (management commands)
• Much slower due to TransactionTestCase

Solution?

django-shareddb: offload all DB operations to a single thread

=> Wrap LiveServerTestCase tests in transactions!

• Allows for a complete test of the whole system
• Provide a ground for failure mode handling

But:

• Rather complex to setup
• Lack of proper tooling
• Kind of giant selenium tests, but with multiple servers and databases

Prototype: fire / djfire.

• Allows remote code execution across virtual environments
• Provides hooks for synchronizing test cases, stopping daemons
• Compatible with django-shareddb

Release: soon!

Example:

class MyRemoteTest(fire_tests.FireTestCase):
    fire_servers = {
        'proj': '../../remote.sock',
    }

    def test_user_factory(self):
        proj = self.fire_remotes['proj']
        auth_factories = proj.module('django_fact.factories')
        auth_models = proj.module('django.contrib.auth.models')

        user = auth_factories.UserFactory()
        user2 = auth_models.User.objects.get()

        self.assertEqual(user, user2)
        self.assertEqual(0, user2.groups.count())