public interface StatsStorage extends StatsStorageRouter
Key design ideas:
(a) Two types of storable objects:
i. Persistable
objects, for once per session objects ("static info") and also for periodically reported data ("updates")
ii. StorageMetaData
objects, for
(b) There are 4 types of things used to uniquely identify these Persistable objects:
i. SessionID: A unique identifier for a single session
ii. TypeID: A unique identifier for the listener or type of data
For example, we might have stats from 2 (or more) listeners with identical session and worker IDs
This is typically hard-coded, per listener class
iii. WorkerID: A unique identifier for workers, within a session
iv. Timestamp: time at which the record was created
For example, single machine training (with 1 listener) would have 1 session ID, 1 type ID, 1 worker ID, and multiple timestamps.
Distributed training multiple listeres could have 1 session ID, multiple type IDs, and multiple worker IDs, and multiple timestamps for each
A hyperparameter optimization job could have multiple session IDs on top of that.
Note that the StatsStorage interface extends StatsStorageRouter
Modifier and Type | Method and Description |
---|---|
void |
close()
Close any open resources (files, etc)
|
void |
deregisterStatsStorageListener(StatsStorageListener listener)
Remove the specified listener, if it is present.
|
java.util.List<Persistable> |
getAllStaticInfos(java.lang.String sessionID,
java.lang.String typeID)
Get all static informations for the given session and type ID
|
java.util.List<Persistable> |
getAllUpdatesAfter(java.lang.String sessionID,
java.lang.String typeID,
long timestamp)
Get all updates for the given session ID (all worker IDs), that occur after (not including) the given timestamp.
|
java.util.List<Persistable> |
getAllUpdatesAfter(java.lang.String sessionID,
java.lang.String typeID,
java.lang.String workerID,
long timestamp)
Get all updates for the given session and worker ID, that occur after (not including) the given timestamp.
|
Persistable |
getLatestUpdate(java.lang.String sessionID,
java.lang.String typeID,
java.lang.String workerID)
Get the latest update record (i.e., update record with the largest timestamp value) for the specified
session and worker IDs
|
java.util.List<Persistable> |
getLatestUpdateAllWorkers(java.lang.String sessionID,
java.lang.String typeID)
Get the latest update for all workers, for the given session ID
|
java.util.List<StatsStorageListener> |
getListeners()
Get a list (shallow copy) of all listeners currently present
|
int |
getNumUpdateRecordsFor(java.lang.String sessionID)
Return the number of update records for the given session ID (all workers)
|
int |
getNumUpdateRecordsFor(java.lang.String sessionID,
java.lang.String typeID,
java.lang.String workerID)
Return the number of update records for the given session ID and worker ID
|
Persistable |
getStaticInfo(java.lang.String sessionID,
java.lang.String typeID,
java.lang.String workerID)
Get the static info for the given session and worker IDs, or null if no such static info has been reported
|
StorageMetaData |
getStorageMetaData(java.lang.String sessionID,
java.lang.String typeID)
Get the session metadata, if any has been registered via
StatsStorageRouter.putStorageMetaData(StorageMetaData) |
Persistable |
getUpdate(java.lang.String sessionID,
java.lang.String typeId,
java.lang.String workerID,
long timestamp)
Get the specified update (or null, if none exists for the given session/worker ids and timestamp)
|
boolean |
isClosed() |
java.util.List<java.lang.String> |
listSessionIDs()
Get a list of all sessions stored by this storage backend
|
java.util.List<java.lang.String> |
listTypeIDsForSession(java.lang.String sessionID)
Get the list of type IDs for the given session ID
|
java.util.List<java.lang.String> |
listWorkerIDsForSession(java.lang.String sessionID)
For a given session ID, list all of the known worker IDs
|
java.util.List<java.lang.String> |
listWorkerIDsForSessionAndType(java.lang.String sessionID,
java.lang.String typeID)
For a given session ID and type ID, list all of the known worker IDs
|
void |
registerStatsStorageListener(StatsStorageListener listener)
Add a new StatsStorageListener.
|
void |
removeAllListeners()
Remove all listeners from the StatsStorage instance
|
boolean |
sessionExists(java.lang.String sessionID)
Check if the specified session ID exists or not
|
putStaticInfo, putStaticInfo, putStorageMetaData, putStorageMetaData, putUpdate, putUpdate
void close() throws java.io.IOException
java.io.IOException
boolean isClosed()
java.util.List<java.lang.String> listSessionIDs()
boolean sessionExists(java.lang.String sessionID)
sessionID
- Session ID to checkPersistable getStaticInfo(java.lang.String sessionID, java.lang.String typeID, java.lang.String workerID)
sessionID
- Session IDworkerID
- worker IDjava.util.List<Persistable> getAllStaticInfos(java.lang.String sessionID, java.lang.String typeID)
sessionID
- Session ID to get static info fortypeID
- Type ID to get static info forjava.util.List<java.lang.String> listTypeIDsForSession(java.lang.String sessionID)
sessionID
- Session ID to queryjava.util.List<java.lang.String> listWorkerIDsForSession(java.lang.String sessionID)
sessionID
- Session IDjava.util.List<java.lang.String> listWorkerIDsForSessionAndType(java.lang.String sessionID, java.lang.String typeID)
sessionID
- Session IDtypeID
- Type IDint getNumUpdateRecordsFor(java.lang.String sessionID)
sessionID
- Session IDint getNumUpdateRecordsFor(java.lang.String sessionID, java.lang.String typeID, java.lang.String workerID)
sessionID
- Session IDworkerID
- Worker IDPersistable getLatestUpdate(java.lang.String sessionID, java.lang.String typeID, java.lang.String workerID)
sessionID
- session IDworkerID
- worker IDPersistable getUpdate(java.lang.String sessionID, java.lang.String typeId, java.lang.String workerID, long timestamp)
sessionID
- Session IDworkerID
- Worker IDtimestamp
- Timestampjava.util.List<Persistable> getLatestUpdateAllWorkers(java.lang.String sessionID, java.lang.String typeID)
sessionID
- Session IDjava.util.List<Persistable> getAllUpdatesAfter(java.lang.String sessionID, java.lang.String typeID, java.lang.String workerID, long timestamp)
sessionID
- Session IDworkerID
- Worker Idtimestamp
- Timestampjava.util.List<Persistable> getAllUpdatesAfter(java.lang.String sessionID, java.lang.String typeID, long timestamp)
sessionID
- Session IDtimestamp
- TimestampStorageMetaData getStorageMetaData(java.lang.String sessionID, java.lang.String typeID)
StatsStorageRouter.putStorageMetaData(StorageMetaData)
sessionID
- Session ID to get metadatvoid registerStatsStorageListener(StatsStorageListener listener)
listener
- Listener to addvoid deregisterStatsStorageListener(StatsStorageListener listener)
listener
- Listener to removevoid removeAllListeners()
java.util.List<StatsStorageListener> getListeners()