Hypercore

Hypercore is a secure, distributed append-only log for sharing large datasets and realtime streams. It supports sparse replication, verified reads, optional block encryption, and session-based workflows for building higher-level local-first and peer-to-peer data structures.

Install

npm i hypercore

Quickstart

import Hypercore from 'hypercore'

const core = new Hypercore('./my-first-core', { valueEncoding: 'utf-8' })
await core.ready()

await core.append('hello')
await core.append('from hypercore')

console.log('length:', core.length)
console.log('first block:', await core.get(0))
console.log('second block:', await core.get(1))

for await (const block of core.createReadStream()) {
  console.log('streamed:', block)
}

await core.close()

API Reference

Constructors and sessions

new Hypercore(storage, [key], [options])

src

Make a new Hypercore instance.

const core = new Hypercore('./directory') // store data in ./directory

User Data is a local-only key/value store with string keys and string or Buffer values. It is not replicated and is useful for peer-specific metadata such as encryption keys or app state.

The manifest controls how a core is authenticated, including version, hash, quorum, signers, prologue, linked, and manifest-level userData. Changing the manifest changes the resulting Hypercore key.

core.session([options])

src

Creates a new Hypercore instance that shares the same underlying core.

• Returns: Hypercore — A new Hypercore session sharing the same underlying storage.
• Throws: SESSION_CLOSED if called on a core that is already closing.

const core = new Hypercore('./atom-example')
await core.ready()

await core.append('block 1')

const atom = core.state.storage.createAtom()
const atomicSession = core.session({ atom })

await core.append('block 2') // Add blocks not using the atom

await atomicSession.append('atom block 2') // Add different block to atom
await atom.flush()

console.log((await core.get(core.length - 1)).toString()) // prints 'atom block 2' not 'block 2'

await core.commit(session, opts = {})

src

Attempt to apply blocks from the session to the core. core must be a default core, aka a non-named session.

• Returns: Promise<{length: number, byteLength: number}\|null> — null if committing failed.
• Throws: INVALID_OPERATION if no database batch was passed, or the tree changed during the batch.

core.snapshot([options])

src

Same as core.session(options), but backed by a storage snapshot so will not truncate nor append.

• Returns: Hypercore — A new read-only Hypercore session locked to the current length.

const snap = core.snapshot()
console.log(snap.length) // frozen at the moment snapshot() was called

Writing and local mutation

await core.append(block, options = {})

src

Append a block of data (or an array of blocks) to the core. Returns the new length and byte length of the core.

• Returns: Promise<{length: number, byteLength: number}> — The new length and byteLength of the core after appending.
• Throws:
  • SESSION_CLOSED if the core has been closed.
  • SESSION_NOT_WRITABLE if the core is not writable.
  • INVALID_OPERATION if the append is inconsistent with the manifest prologue.
  • BAD_ARGUMENT if an appended block exceeds the maximum suggested block size.

// simple call append with a new block of data
await core.append(Buffer.from('I am a block of data'))

// pass an array to append multiple blocks as a batch
await core.append([Buffer.from('batch block 1'), Buffer.from('batch block 2')])

core.createWriteStream()

src

Make a write stream to append chunks as blocks.

• Returns: object — A Writable stream; each chunk written becomes one block appended to the core.

const ws = core.createWriteStream()

// Listen for stream finishing
const done = new Promise((resolve) => ws.on('finish', resolve))

for (const data of ['hello', 'world']) ws.write(data)
ws.end()

await done

console.log(await core.get(core.length - 2)) // 'hello'
console.log(await core.get(core.length - 1)) // 'world'

await core.clear(start, [end], [options])

src

Clear stored blocks between start and end, reclaiming storage when possible.

• Returns: Promise<{blocks: number}\|null> — { blocks } with the count of cleared blocks when opts.diff is true, otherwise null.
• Throws:
  • SESSION_CLOSED if the core has been closed.
  • ASSERTION if the start/end range is invalid.

await core.clear(4) // clear block 4 from your local cache
await core.clear(0, 10) // clear block 0-10 from your local cache

await core.truncate(newLength, [options])

src

Truncate the core to a smaller length.

• Returns: Promise<void> — Resolves once the truncation has been signed and written to storage.
• Throws:
  • SESSION_CLOSED if the core has been closed.
  • SESSION_NOT_WRITABLE if the core is not writable.
  • INVALID_OPERATION if the truncation would break the manifest prologue.

Reading, streams, and proofs

await core.get(index, [options])

src

Get a block of data. If the data is not available locally this method will prioritize and wait for the data to be downloaded.

• Returns: Promise<Buffer\|null> — The block value (decoded per valueEncoding), or null if the block is not available and wait is false.
• Throws:
  • SESSION_CLOSED if the core has been closed.
  • ASSERTION if index is not a valid block index.

// get block #42
const block = await core.get(42)

// get block #43, but only wait 5s
const blockIfFast = await core.get(43, { timeout: 5000 })

// get block #44, but only if we have it locally
const blockLocal = await core.get(44, { wait: false })

await core.has(start, [end])

src

Check if the core has all blocks between start and end.

• Returns: Promise<boolean> — true if every block in [start, end) is available locally.
• Throws: ASSERTION if the start/end range is invalid.

const hasBlock = await core.has(5)
const hasRange = await core.has(0, 10)

await core.update([options])

src

Waits for initial proof of the new core length until all findingPeers calls have finished.

• Returns: Promise<boolean> — true if the core was updated to a new length, false otherwise.

const updated = await core.update()

console.log('core was updated?', updated, 'length is', core.length)

const [index, relativeOffset] = await core.seek(bytes, [options])

src

Seek to a byte offset.

• Returns: Promise<[number, number]> — [index, relativeOffset], where index is the data block the bytes is contained in and relativeOffset is the relative byte offset in the data block.
• Throws:
  • SESSION_CLOSED if the core has been closed.
  • ASSERTION if bytes is not a valid byte offset.

await core.append([Buffer.from('abc'), Buffer.from('d'), Buffer.from('efg')])

const first = await core.seek(1) // returns [0, 1]
const second = await core.seek(3) // returns [1, 0]
const third = await core.seek(5) // returns [2, 1]

core.createReadStream([options])

src

Make a read stream to read a range of data out at once.

• Returns: object — A Readable stream that emits decoded blocks.

// read the full core
const fullStream = core.createReadStream()

// read from block 10-14
const partialStream = core.createReadStream({ start: 10, end: 15 })

// pipe the stream somewhere using the .pipe method on Node.js or consume it as
// an async iterator

for await (const data of fullStream) {
  console.log('data:', data)
}

core.createByteStream([options])

src

Make a byte stream to read a range of bytes.

• Returns: object — A Readable stream that emits raw Buffer chunks spanning the requested byte range.

// Read the full core
const fullStream = core.createByteStream()

// Read from byte 3, and from there read 50 bytes
const partialStream = core.createByteStream({ byteOffset: 3, byteLength: 50 })

// Consume it as an async iterator
for await (const data of fullStream) {
  console.log('data:', data)
}

// Or pipe it somewhere like any stream:
partialStream.pipe(process.stdout)

await core.treeHash([length])

src

Get the Merkle Tree hash of the core at a given length, defaulting to the current length of the core.

• Returns: Promise<Buffer> — A 32-byte BLAKE2b hash of the Merkle tree roots at the given length.

const hash = await core.treeHash()
console.log(hash.toString('hex'))

await core.proof(opts)

src

Generate a proof (a TreeProof instance) for the request opts.

• Returns: Promise<object> — A settled TreeProof object that can be serialised and sent to a remote peer.

await core.verifyFullyRemote(proof)

src

Note that you cannot seek & provide a block / hash request when upgrading.

• Returns: Promise<object> — the merkle tree batch from the proof.

const batch = await core.verifyFullyRemote(remoteProof)

await core.signable([length], [fork])

src

Produce the signable payload for a given tree state. The payload encodes the core's key, tree hash (core.treeHash()), length, and fork.

• Returns: Promise<Buffer> — a buffer which encodes the core's key, tree hash (core.treeHash()), length, & fork.

const payload = await core.signable()
const sig = sodium.crypto_sign_detached(payload, secretKey)

core.download([range])

src

Download a range of data.

• Returns: object — A Download handle with a .done() promise and a .destroy() method to cancel.

const download = core.download({ start: 0, end: 10 })
await download.done()

// Note that this will never be considered downloaded as the range
// will keep waiting for new blocks to be appended.
core.download({ start: 0, end: -1 })

core.download({ blocks: [4, 9, 7] })

// will stop downloading now
range.destroy()

Extensions and replication

core.registerExtension(name, handlers = {})

src

Register a custom protocol extension. This is a legacy implementation and is no longer recommended. Creating a Protomux protocol is recommended instead.

• Returns: object — The extension object with send(message, peer), broadcast(message), and destroy() methods.

This extension API is documented as legacy. For new protocol work, the upstream README recommends creating a Protomux protocol instead.

ext.send(message, peer)

Sends the message to a specific peer.

ext.broadcast(message)

Sends the message to all peers.

ext.destroy()

Unregister and remove extension from the hypercore.

core.replicate(isInitiatorOrReplicationStream, opts = {})

src

Create a replication stream. You should pipe this to another Hypercore instance.

• Returns: object — The replication stream (a raw duplex stream with a Protomux attached).

// assuming we have two cores, localCore + remoteCore, sharing the same key
// on a server
const net = require('net')
const server = net.createServer(function (socket) {
  socket.pipe(remoteCore.replicate(false)).pipe(socket)
})

// on a client
const socket = net.connect(...)
socket.pipe(localCore.replicate(true)).pipe(socket)

core.findingPeers()

src

Create a hook that tells Hypercore you are finding peers for this core in the background. Call done when your current discovery iteration is done. If you're using Hyperswarm, you'd normally call this after a swarm.flush() finishes.

• Returns: function — A done() callback to call when peer discovery is complete.

const done = core.findingPeers()
swarm.flush().then(done, done)

Storage inspection and mark-and-sweep

await core.info([options])

src

Get information about this core, such as its total size in bytes.

• Returns: Promise<object> — An Info object with key, discoveryKey, length, contiguousLength, byteLength, fork, padding, and optional storage fields.

Info {
  key: Buffer(...),
  discoveryKey: Buffer(...),
  length: 18,
  contiguousLength: 16,
  byteLength: 742,
  fork: 0,
  padding: 8,
  storage: {
    oplog: 8192,
    tree: 4096,
    blocks: 4096,
    bitfield: 4096
  }
}

await core.startMarking()

src

This enables marking mode for the "mark & sweep" approach to clear hypercore storage. When called the current markings are cleared.

• Returns: Promise<void> — Resolves once marking mode is active and previous marks are cleared.
• Throws: ASSERTION if the core is already in gc mode, or is a named or atomic session.

await core.startMarking()
await core.get(2)
await core.get(4)
await core.sweep() // All blocks but blocks 2 & 4 are cleared

await core.markBlock(start, end = start + 1)

src

Manually mark a block or range of blocks to be retained when sweeping. Useful to mark blocks without loading them into memory. end is non-inclusive and defaults to start + 1 so core.markBlock(index) only marks the block at index.

• Returns: Promise<void> — Resolves once all marks in the range have been written to storage.

await core.markBlock(5)       // mark block 5
await core.markBlock(0, 10)   // mark blocks 0–9

await core.clearMarkings()

src

Manually remove all markings. Automatically called when calling core.startMarking().

• Returns: Promise<void> — Resolves once all marks have been cleared from storage.

await core.clearMarkings()

await core.sweep(opts)

src

Clear all unmarked blocks from storage.

• Returns: Promise<void> — Resolves once all unmarked blocks have been cleared.

The README describes the full mark-and-sweep flow as startMarking(), read or manually mark the blocks you want to keep, then sweep() to clear everything else.

Lifecycle and local configuration

await core.close([{ error }])

src

Fully close this core. Passing an error via { error } is optional and all pending replicator requests will be rejected with the error.

• Returns: Promise<void> — Resolves once the session (and underlying core if no other sessions remain) is fully closed.

await core.close()

await core.ready()

src

Wait for the core to fully open.

• Returns: Promise<void> — Resolves once the core is ready for reading and writing.

const core = new Hypercore('./storage')
await core.ready()
console.log(core.key) // now available

await core.setEncryption(encryption)

src

Set the encryption, which should satisfy the HypercoreEncryption interface.

• Returns: Promise<void> — Resolves once the encryption provider has been installed.
• Throws: ASSERTION if the provider does not satisfy the HypercoreEncryption interface.

await core.setEncryption(new DefaultEncryption(encryptionKey, core.key))

await core.setEncryptionKey(key, [opts])

src

Set the encryption key.

• Returns: Promise<void> — Resolves once the encryption provider has been installed.

core.setKeyPair(keyPair)

src

Update the core's keyPair. Advanced as the keyPair is used throughout Hypercore, e.g. verifying blocks, identifying the core, etc.

• Returns: void

core.setKeyPair({ publicKey: pubKey, secretKey: secKey })

core.setActive(active)

src

Set the core to be active or not. A core is considered 'active' if it should linger to download blocks from peers.

• Returns: void

core.setActive(false) // stop lingering for peer downloads

await core.setGroup(topic)

src

Set the group topic that the hypercore belongs to. Useful for grouping hypercores together that need to update a larger data structure (eg. autobee) that is comprised of them. See corestore's store.notifyGroup(topic) for more details.

await core.setUserData(key, value)

src

Set a key in the User Data key-value store.

• Returns: Promise<void> — Resolves once the value has been persisted to storage.

await core.setUserData('version', Buffer.from('1.0'))

await core.getUserData(key)

src

Reads the local user-data value stored under key, resolving with its Buffer/string value or null if unset. User data is local-only and not replicated.

• Returns: Promise<Buffer\|null> — the value for a key in the User Data key-value store.

const value = await core.getUserData('version')
console.log(value && value.toString()) // '1.0'

Properties

core.writable

src

Can we append to or truncate this core?

• Returns: boolean

core.readable

src

Can we read from this core? After closing the core this will be false.

• Returns: boolean

core.id

src

String containing the id (z-base-32 of the public key) identifying this core.

• Returns: string\|null — The z-base-32 encoded public key, or null before the core is opened.

core.key

src

Buffer containing the public key identifying this core.

• Returns: Buffer\|null — The 32-byte public key, or null before the core is opened.

core.keyPair

src

Object containing buffers of the core's public and secret key

• Returns: {publicKey: Buffer, secretKey: Buffer\|null}\|null

core.discoveryKey

src

Buffer containing a key derived from the core's public key. In contrast to core.key this key does not allow you to verify the data but can be used to announce or look for peers that are sharing the same core, without leaking the core key.

• Returns: Buffer\|null — The 32-byte discovery key, or null before the core is opened.

core.length

src

How many blocks of data are available on this core.

• Returns: number — The total number of blocks in the core (0 before the core is opened).

core.signedLength

src

How many blocks of data are available on this core that have been signed by a quorum. This is equal to core.length for Hypercores with a single signer.

• Returns: number — The number of quorum-signed blocks (0 before the core is opened).

core.contiguousLength

src

How many blocks are contiguously available starting from the first block of this core.

• Returns: number — The local contiguous length (0 before the core is opened).

core.remoteContiguousLength

src

How many blocks are contiguously available starting from the first block of this core on any known remote. This is only updated when a remote thinks it is fully contiguous such that they have all known blocks.

• Returns: number — The remote contiguous length (0 before the core is opened).

core.fork

src

What is the current fork id of this core?

• Returns: number — The current fork counter (incremented on each truncation; 0 before the core is opened).

core.padding

src

How much padding is applied to each block of this core? Will be 0 unless block encryption is enabled.

• Returns: number — Number of padding bytes prepended to each block (0 when encryption is disabled).

core.peers

src

Array of current peers the core is replicating with.

• Returns: Array<object> — Live array of Replicator peer objects (empty before the core is opened).

Events

core.on('close')

src

Emitted when the core has been fully closed.

core.on('ready')

src

Emitted after the core has initially opened all its internal state.

core.on('append')

Emitted when the core has been appended to (i.e. has a new length / byteLength), either locally or remotely.

core.on('truncate', ancestors, forkId)

Emitted when the core has been truncated, either locally or remotely.

core.on('peer-add')

Emitted when a new connection has been established with a peer.

core.on('peer-remove')

Emitted when a peer's connection has been closed.

core.on('upload', index, byteLength, peer)

Emitted when a block is uploaded to a peer.

core.on('download', index, byteLength, peer)

Emitted when a block is downloaded from a peer.

core.on('remote-contiguous-length', length)

Emitted when the max known contiguous length from a remote, ie core.remoteContiguousLength, is updated. Note this is not emitted when core is truncated.

Static helpers

Hypercore.MAX_SUGGESTED_BLOCK_SIZE

src

The constant for max size (15MB) for blocks appended to Hypercore. This max ensures blocks are replicated smoothly.

• Returns: number

Hypercore.key(manifest, options = {})

src

options include:

• Returns: Buffer — the key for a given manifest.

Hypercore.discoveryKey(key)

src

Derive the discovery key from a Hypercore public key. The discovery key can safely be shared to announce the core without exposing the read key.

• Returns: Buffer — the discovery key for the provided key.

const dKey = Hypercore.discoveryKey(core.key)

Hypercore.blockEncryptionKey(key, encryptionKey)

src

Derive a block-level encryption key from the Hypercore public key and a master encryption key using BLAKE2b.

• Returns: Buffer — a block encryption key derived from the key and encryptionKey.

const blockKey = Hypercore.blockEncryptionKey(core.key, encryptionKey)

Hypercore.getProtocolMuxer(stream)

src

Extract the Protomux instance attached to a Hypercore protocol stream.

• Returns: object — a protomux instance from the provided stream Hypercore protocol stream.

const stream = core.replicate(true)
const mux = Hypercore.getProtocolMuxer(stream)

Hypercore.createCore(storage, opts)

src

Create the raw internal Core object directly, bypassing the Hypercore session layer. Useful for low-level tooling that needs direct storage access.

• Returns: object — the internal core using the storage and opts without creating a full Hypercore instance.

const core = Hypercore.createCore('./storage', { key: myKey })

Hypercore.createProtocolStream(isInitiator, opts = {})

src

Create an encrypted noise stream with a protomux instance attached used for Hypercore's replication protocol.

• Returns: object — The outer raw stream with a .noiseStream.userData Protomux attached.

Hypercore.defaultStorage(storage, opts = {})

src

Wrap a path or existing CoreStorage object into a CoreStorage instance. Called internally by the constructor; exposed for advanced use.

• Returns: object — a default hypercore storage.

const storage = Hypercore.defaultStorage('./my-core')

Types

HypercoreOptions

Options for creating or opening a Hypercore instance.

KeyOptions

Options for Hypercore.key() static method.

ProtocolStreamOptions

Options for Hypercore.createProtocolStream().

DefaultStorageOptions

Options for Hypercore.defaultStorage().

SessionOptions

Options for core.session() / core.snapshot().

SetEncryptionKeyOptions

Options for core.setEncryptionKey().

GetOptions

Options for core.get().

SeekOptions

Options for core.seek().

ClearOptions

Options for core.clear().

UpdateOptions

Options for core.update().

TruncateOptions

Options for core.truncate().

AppendOptions

Options for core.append().

InfoOptions

Options for core.info().

ReadStreamOptions

Options for core.createReadStream().

ByteStreamOptions

Options for core.createByteStream().

DownloadRange

Download range descriptor for core.download().

ProofOptions

Options for core.proof().

CommitOptions

Options for core.commit().

SweepOptions

Options for core.sweep().

CloseOptions

Options for core.close().

Errors

Coded errors this module can throw — catch them via err.code.

See also

• Replicate and persist with Hypercore—pair a single core with peer discovery and replication.
• Work with many Hypercores using Corestore—recommended pattern when one process manages multiple cores.
• Corestore—helper reference for creating, naming, and co-replicating groups of Hypercores.
• Hyperbee—sorted key/value B-tree built directly on top of a Hypercore.
• Hyperdrive—filesystem abstraction whose metadata and blob stores are Hypercore-backed.
• Compact encoding—binary encoding toolkit used to structure Hypercore block payloads.