Elite: Dangerous e CosmosDB

o7 cmdr!

Em uma noite quente de quarentena, em um dos telegramas de bate-papo no Elite: Dangerous, surgiu uma discussão sobre o tema: que tipo de estrelas têm planetas semelhantes à Terra com mais frequência?

O fato é que a exploração de planetas é uma das principais mecânicas do jogo. E na hierarquia da utilidade dos planetas, os semelhantes à Terra estão no topo. Mas sua raridade também é bastante alta. Então, os comandantes queriam saber: em quais estrelas prestar atenção ao se mover pela Via Láctea?

Dessa discussão nasceu todo um projeto, que acabei enterrando. Não, encontramos a resposta para a pergunta feita com ele. Mas não gostei do projeto por vários motivos e, depois de alguns meses de procrastinação, lancei a segunda iteração. O que resultou disso, bem como a resposta à pergunta feita - neste artigo.

Introdução

Na elite, uma grande parte da jogabilidade acontece, não importa o que você pense, em um navegador com uma dúzia de abas abertas. Não vamos argumentar corretamente ou não, mas, definitivamente, isso tem seu próprio charme e muitas pessoas gostam disso.

, , . , Frontier Developments . , .

? , Log- . .

() ( Windows):

C:\Users\User Name\Saved Games\Frontier Developments\Elite Dangerous\

. . .. PC ( ) — .

, , , . ? - , ( , , , , POI, etc).

PC:

. - . , . , , . , 400 (, ! ). , -.

, . ( , ). : , , (, ) .. , , , , — EDSM EDDB.

, , — . .

EDSM , . , , . , , — JSON 10. - - . , , - , . , ! .

% . 6 . .

, , . , , !

, , , - ( , ). EDSM, EDDB, etc. — EDDN. , , , .

, EDDN ? ...

EDDN

EDDN — :

The Elite: Dangerous Data Network is a system for willing Commanders to share dynamic data about the galaxy with others.

By pooling data in a common format, tools and analyses can be produced that add an even greater depth and vibrancy to the in-game universe.



EDDN is not run by or affiliated with Frontier Developments.

( ).

HTTP Endpoint https://eddn.edcd.io:4430/upload/ ( ).

. ZeroMQ tcp://eddn.edcd.io:9500. .

( ):

, . ? , . , ( ) ! , - .

, :

— Microsoft Azure. - . — Azure, dotnet core/standard

--- , .

, :

1. Message Distributor

EDDN Channel . ( schemaRef , ) — . EDDN 5:

• https://eddn.edcd.io/schemas/journal/1
• https://eddn.edcd.io/schemas/blackmarket/1
• https://eddn.edcd.io/schemas/commodity/3
• https://eddn.edcd.io/schemas/outfitting/2
• https://eddn.edcd.io/schemas/shipyard/2

. ? , ( ), , . journal, .. ( ). , , .

, MessageDistributor EDDN . .

2. Azure Storage Queue

( Message Processor , ). , . Storage Account ( ) connection string ( , Azure AD, ). Storage Account ( — , , ).

Azure Storage Emulator Azure Storage Explorer

3. Message Processor

, , . Azure Function App.

, - ? .

, Azure Function WebJob ( , ), . , Azure Function Runtime ( , ) — — , — . , .. — .

, , (Scale Out), , , - , CPU/Memory consumption, etc.

, (in/out bindings). , , QueueTrigger — (, dotnet, ). CosmosDBTrigger . ( ). , , CosmosDB ( ) . - , , : db client ( in-built DI, ). , , queue .

QueueTrigger. , — (invisible) . , 30 ( ). 30 — DequeueCount 1. ( -> visible state on, ++DequeueCount). DequeueCount = 5 ( ), -poison. journal, 5 , journal-poison. ( - ). , . .

: WebJob AppService. — Function App. : , .

. : , , 2 . 2 , , . , (upsert — update || insert) , , ;-) 2 , , . , . , . , , , . , - (, , CosmosDB ACID compliant).

IMHO: , . . 1. , — . (, ) .

Cosmos DB Optimistic Concurrency, . . .

4. CosmosDB

. , ( ) , . , .. .

4: Signals, Systems, Stations, Bodies. , journal ( , ? -).

, journal Event, : CarrierJump, Docked, FsdJump, Location, SaaSignalsFound, Scan ( ). , — . : — . .

CosmosDB. ( , ) Request Unit per second — RU/s. .. RU/s . 2 : Provisioned throughput Serverless ( , ). , Provisioned RU/s ( ), Serverless, RU/s , , .



Provisioned RU/s. , RU/s 400, 10% 1000% (40-4000 RU/s)



RU/s Provisioned mode — 400. , , , 250 RU/s 150 , Serverless .



, ( CosmosDB). : CosmicClone. Serverless . 2 . Provisioned Mode , , .



, zero downtime "" — , MessageDistributor , . , . . .



CosmosDB

, , : EDDN , .

, React Native — dotnet, . .

: EDDNConsumer. , , .

:

• EDDNConsumerCore — MessageDistributor. EDDN
• EDDNModels — EDDN
• JournalContributor — MessageProcessor journal
• SharedLibrary — . .

EDDNConsumerCore

dotnet core 3.1 . Main DI. HostedService ConsumerService

services.AddHostedService<ConsumerService>();

nuget- ( ): NetMQ — ZeroMq Ionic.Zlib — NetMQ

StartAsync, — NetMQRuntime ClientAsync ( , ):

private async Task ClientAsync()
{
    var utf8 = new UTF8Encoding();
    using (var client = new SubscriberSocket())
    {
        client.Connect(_eddnClientSettings.ConnectionString);
        client.SubscribeToAnyTopic();
        while (true)
        {
            try
            {
                (var bytes, _) = await client.ReceiveFrameBytesAsync();
                var uncompressed = ZlibStream.UncompressBuffer(bytes);
                var result = utf8.GetString(uncompressed);
                await _messageDistributor.DistributeAsync(result);
                _logger.LogInformation(result);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error reading message queue");
            }
        }
    }
}

, ( , 1 ), _messageDistributor , , . .

MessageDistributor DistributeAsync:

public async Task DistributeAsync(string message)
{
    try
    {
        using var stringReader = new StringReader(message);
        using var jsonReader = new JsonTextReader(stringReader);
        var result = _serializer.Deserialize<Entity<BaseMessage>>(jsonReader);
        var queue = await _messageQueueFactory.GetQueueAsync(result);
        await queue.SendMessageAsync(message.Base64Encode());
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Error distributing message");
    }
}

, , MessageQueueFactory. — . : . , schemaRef header ( test_data.json ). , , .

, ( — JSON) , . \\ , — , , null default. ( ) , POCO , , , , - . .

, , json- NewtonsoftJson ( json, ). Missing Member

_serializer.MissingMemberHandling = MissingMemberHandling.Error;
_serializer.Error += _serializer_Error;

, Application Insights, Notification Alerts . , , Header. . , .. , missing property. , , .

, header , . , ( ):

public class MessageQueueFactory : IMessageQueueFactory
{
    private readonly StorageAccount _storageOptions;
    private readonly QueueMapping _queueMapping;
    private readonly IDictionary<string, QueueClient> _queues = new Dictionary<string, QueueClient>();

    public MessageQueueFactory(
        IOptions<StorageAccount> storageOptions,
        IOptions<QueueMapping> queueMapping)
    {
        _storageOptions = storageOptions.Value;
        _queueMapping = queueMapping.Value;
    }

    public async Task<QueueClient> GetQueueAsync(Entity<BaseMessage> entity)
    {
        if (_queueMapping.TryGetValue(entity.SchemaRef, out var queueName))
        {
            if (!_queues.ContainsKey(queueName))
            {
                var client = new QueueClient(_storageOptions.StorageConnectionString, queueName);
                await client.CreateIfNotExistsAsync();
                _queues.Add(queueName, client);
            }
            return _queues[queueName];
        }
        throw new ArgumentException($"Queue {entity.SchemaRef} has not configured", nameof(entity.SchemaRef));
    }
}

, appsettings json :

{
  "QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal"
  }
}

( ):

"QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal",
    "eddn.edcd.io/schemas/blackmarket/1": "blackmarket",
    "eddn.edcd.io/schemas/commodity/3": "commodity",
    "eddn.edcd.io/schemas/outfitting/2": "outfitting",
    "eddn.edcd.io/schemas/shipyard/2": "shipyard"
  }

Dictionary<string, string> schemaRef. .

1 _queues — QueueClient, . , . , . round trip , . , , . , .

, — , — . . MessageDistributor, .

. 3 . .

EDDNModels

POCO . , . , JournalMessage:

[JsonProperty("id")]
public override string Id
{
    get => Event switch
    {
        JournalEvent.FsdJump => StarSystem,
        JournalEvent.Scan => BodyName,
        JournalEvent.Docked => StationName,
        JournalEvent.Location => BodyName,
        JournalEvent.CarrierJump => StarSystem,
        JournalEvent.SaaSignalsFound => BodyName,
        _ => $"UnknownID_{Guid.NewGuid()}"
    };
}

Id, , CosmosDB. ORM. , journal ? , . Event Id. - , - , - .

. : , , , , , , , , , \ .. ( JournalMessage). , , .. POCO — .

.

JournalContributor

"" journal — Azure Function App. Function App dotnet core class library , , . FunctionName . : — Function App ( ). - — . , . . , , ( ).

, :

public class JournalContributor
    {
        private readonly IEventTypeProcessorFactory _eventTypeProcessorFactory;

        public JournalContributor(IEventTypeProcessorFactory eventTypeProcessorFactory)
        {
            _eventTypeProcessorFactory = eventTypeProcessorFactory;
        }

        [FunctionName("ContributeJournal")]
        public async Task Run(
            [QueueTrigger("journal", Connection = "AzureWebJobsStorage")]
            Entity<JournalMessage> myQueueItem,
            ILogger log)
        {
            try
            {
                var eventProcessor = _eventTypeProcessorFactory.GetProcessor(myQueueItem.Message);
                await eventProcessor.ProcessEventAsync(myQueueItem.Message);
            }
            catch(Exception ex)
            {
                log.LogError(ex, $"Error processing queue item: {JsonConvert.SerializeObject(myQueueItem)}");
                throw;
            }
        }
    }

QueueTrigger ( ). , Event … .

, , . — , dotnet . Startup Configure, FunctionStartup (assembly):

using JournalContributor.Settings;
using Microsoft.Extensions.Configuration;
using System.IO;

[assembly: FunctionsStartup(typeof(JournalContributor.Startup))]

namespace JournalContributor
{
    public class Startup : FunctionsStartup
    {
        private IConfigurationRoot _functionConfig;
        private readonly string COSMOS_CONNECTION_STRING = Environment.GetEnvironmentVariable("CosmosDBConnectionString");
        private readonly string ENVIRONMENT = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        public override void Configure(IFunctionsHostBuilder builder)
        {
            _functionConfig = new ConfigurationBuilder()
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: true)
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, $"appsettings.{ENVIRONMENT}.json"), optional: true, reloadOnChange: true)
                .Build();

            builder.Services.AddSingleton<CosmosClient>(factory => new CosmosClient(COSMOS_CONNECTION_STRING));
            builder.Services.AddSingleton<IEventTypeProcessorFactory, EventTypeProcessorFactory>();
            builder.Services.AddTransient<FsdJumpProcessor>();
            builder.Services.AddTransient<ScanProcessor>();
            builder.Services.AddTransient<DockedProcessor>();
            builder.Services.AddTransient<LocationProcessor>();
            builder.Services.AddTransient<SaaSignalsFoundProcessor>();
            builder.Services.Configure<CosmosDbSettings>(_functionConfig.GetSection("CosmosDbSettings"));
        }
    }
}

DI . , CosmosClient : (CosmosDB). ? . 2 CosmosDB — , . , .

. , , , switch :

public IEventTypeProcessor GetProcessor(JournalMessage journalMessage) => journalMessage.Event switch
        {
            JournalEvent.FsdJump => _serviceProvider.GetService<FsdJumpProcessor>(),
            JournalEvent.Scan => _serviceProvider.GetService<ScanProcessor>(),
            JournalEvent.Docked => _serviceProvider.GetService<DockedProcessor>(),
            JournalEvent.Location => _serviceProvider.GetService<LocationProcessor>(),
            JournalEvent.SaaSignalsFound => _serviceProvider.GetService<SaaSignalsFoundProcessor>(),
            JournalEvent.CarrierJump => _serviceProvider.GetService<DockedProcessor>(),
            _ => throw new ArgumentException($"Unknown Event {journalMessage.Event}", nameof(journalMessage.Event))
        };

, , . ScanProcessor, .. 2- :

public async Task ProcessEventAsync(JournalMessage message)
        {
            var existingItem = await message.CheckIfItemExists(_bodiesContainer, _options.BodiesCollection.PartitionKey);
            if (existingItem == null)
                await _bodiesContainer.UpsertItemAsync(message);
            else
            {
                //Basic scan type contains less information then others.
                //We`re skipping item upsert if remote item has scan type higher then Basic
                //We`re also skippings if remote item scan type is Detailed (as it`s a maximum info scan)
                //Will update item if both (remote and current) have scan type Basic
                //And will upsert item in case of current item have higher scan type then remote
                if ((message.ScanType == ScanType.Basic && existingItem.ScanType != ScanType.Basic) ||
                    existingItem.ScanType == ScanType.Detailed)
                    return;
                else
                    await _bodiesContainer.UpsertItemAsync(message);
            }
        }

, , ( !) , , , , . . . , . , : , Bodies Id . , .

UpSert, Insert, .. concurrency : Item with such Id already exists ( - ). , , Optimistic Concurrency . . upsert.

, ScanType. Detailed — . Basic — — . Basic — . .

. , 3 , MessageProcessor journal .

, . , Azure WebJob EDDN , Storage Account , Azure Function App Cosmos DB . Application Insights (, -, , ). , .

continuous, .. .

![image](img src="https://habrastorage.org/webt/i_/5u/zq/i_5uzqxkanblmotxacqhtwnp_wq.png)

, , schemaRef . , .

Stream-log : , , … .

. , , .

RU/s . — Provisioned Mode 400 RU/s. , . ~105RU/s.

. 145 . EDSM, .

. Check.

. . .

. . - .

45 . , MSDN , 45 . . , . . 45 — 1.5 . , — .

— . - — , - — "" . .

. ~10 . appinsjournalweprivate . Function App. — , . \. :

( ), host.json JournalContributor:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    },
    "logLevel": {
      "default": "Information",
      "Host": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  }
}

:

, ( 21 ):

( 193 ). - / . 21 1.75 . — appinsweprivate — WebJob. 0.98 0.19 21 .

cdbweprivate — CosmosDB Provisioned Mode 400 RU/s. . cdbslweprivate — Serverless mode CosmosDB. 4.47 .

— laweprivate. Log Analytics Workspace, , , .. CosmosDB. , , , . .

, 175,8 . , .

, . -. appinsights — 3 . , . , % , . - , , 3 — ( ).

-. CosmosDB RU/s ( , ) RU/s (Pricing), :

EDDN. , , Elite: Dangerous. - " " — . , , DLC Odyssey , . .

, Function App Storage Account, , .

, . — .

. .

.

" " " ", . - , . =) , -, ...

— . , - , 145 , EDSM 57 . - - , .

, , , . . " " — .

, — . ? . , , EDSM . , JSON — . — ( ). RU/s , , . .

CosmosDB — Jupyter Notebooks. C#, Python, - . , .

, , . ( , Sol [0,0,0]:

? :

5 :

EDSM, - , . , , — - . , , .

CosmosDB, Azure Function Change Feed ( ), .

— CosmosDB .

Para aqueles que resistiram até o fim e ainda estão interessados ​​em onde procurar planetas semelhantes à Terra na elite, aqui está sua resposta (os resultados são baseados em dados obtidos de um despejo EDSM completo há cerca de seis meses):

Os tipos F, K, G, A e, por algum motivo, nêutrons constituem os 5 principais tipos de estrelas a partir das quais os pilotos encontraram planetas semelhantes à Terra. Bem, estes são os resultados ¯ \ _ (ツ) / ¯

Bem. Isso é tudo.

Voe seguro, cmdr!