Producers are data-generating applications, such as sensors in automobiles or activity
loggers in servers. Producers create messages with the collected data and publish the messages
to HPE Ezmeral Data Fabric Event Store topics, specifically, to HPE Ezmeral Data Fabric Event Store topic-partitions.
Permissions
Before a producer can publish to topics, the user ID running the producer needs these
permissions:
- The
writeAce permission on the volume where the streams are located.
For information about how to set
permissions on volumes, see Setting Whole Volume ACEs.
- The
produceperm permission on the streams where the topics are located.
Users with the adminperm permission on those streams can grant the
produceperm permission.
Producing Messages
Producers create messages about the collected data and send the collected data to a
HPE Ezmeral Data Fabric Event Store producer client
library. In addition to the actual message, the producer specifies the topic that the
message is intended for and an optional partition ID. The producer client buffers incoming
messages and sends them (in batches) to the
HPE Ezmeral Data Fabric Event Store server.
Note: In case of server failure, the producer client automatically continues to retry
sending messages.
Attention: As of data-fabric 6.1, the HPE Ezmeral Data Fabric Event Store API enforces a
maximum of 4096 partitions for a topic. That is, when you create an application with the
API, the maximum number of partitions is 4096. If you previously created an application with
HPE Ezmeral Data Fabric Event Store 6.0.1 API (or older) and you have upgraded,
the original number of partitions can be used. For example, if you were using more than 4096
partitions in data-fabric 6.0.1 or
earlier, you will be able to continue with the same number of partitions after
upgrading.
Event-time Timestamp
As of data-fabric 6.0.1, HPE Ezmeral Data Fabric Event Store supports an event-time timestamp. The timestamp type
can be either createtime (default) or logappendtime. See
the maprcli
stream create and stream edit for more information about these
parameters.
Tip: Since each message is automatically published into a topic-partition with an
event-time timestamp as part of the message record, this allows the Consumer application to
seek records based on the timestamp.
Idempotent (exactly once)
Producers
An "exactly-once" message delivery semantic produces messages without
duplication. Each message is delivered once and only once. Exactly-once is insured by uniquely
identifying a group of messages that are atomically persisted. Exactly-once message delivery
is set with the producer idempotence option. See
Modes of Publishing for more information.
The following failure scenarios are addressed with idempotence:
- The stream processor might take input from multiple source topics and the ordering
across these source topics is not deterministic across multiple runs. So if you re-run
your stream processor that takes input from multiple source topics, it might produce
different results.
- The stream processor might produce output to multiple destination topics. If the
producer cannot do an atomic write across multiple topics, then the producer output can
be incorrect if writes to some (but not all) partitions fail.
- The stream processor might aggregate or join data across multiple inputs. If one of
the instances of the stream processor fails, then you need to be able to rollback the
state materialized by that instance of the stream processor. On restarting the instance,
you also need to be able to resume processing and recreate its state.
- The stream processor might look up enriching information in an external database or by
calling out to a service that is updated out of band. By depending on an external
service, the stream processor can be fundamentally non-deterministic. For example, if
the external service changes its internal state between two runs of the stream
processor, it can lead to incorrect results downstream.