Android OS library

[TimLoong]

Request

I want to try using CLP on Android，for compressing log on Android

Possible implementation

provide a library like jar，or program for android 64bit os. tks

[diy1]

Thanks for the request. Could you share more details about your use case? Specifically:

• Do you care more about compression efficiency (i.e., memory usage, speed), or compression ratio?
• What's the use case of the logs? Are they being searched? Or mostly viewed on a per-file basis in a text editor?
• Do you need any real-time alerting during compression (e.g., generate an alert if a certain pattern is encountered)?

We ask because we could have different designs of the compression library that explore different trade-offs.

We are happy to schedule a meeting to discuss further.

[bdchello]

The current moderate increase in memory and CPU is acceptable, and how much can the compression ratio be improved?

[chengquanzhao]

I Have same need as well.
FOR ME

• I care more about compression ratio
• Logs are generally used for search and for developers to view and analyze textual data
• I do not need any real-time alerting during compression

[TimLoong]

@diy1 like @chengquanzhao said

• I hope low memory and low cpu costs ,higher compression ratio is perfect.Because I need low-power consumption and saving more logs on devices.
• I want searching log,
• I don't care about any real-time alerting during compression.
  tks.

[diy1]

How about we package CLP's compression into a logging library that provides the same APIs as the Android's standard logging libraries (android.util.Log)? That way app developers only need to import our logging library without changing the actual logging statements.

Ideally we’d implement CLP's compression as a plugin for the logging library, like what we did for Python, Go, and Log4j which were deployed at Uber. In those cases, applications don't even need to change the import, but it looks like ART's logging library doesn't support plugins (so we can't do it unless you're building Android yourself).

Or are you trying to compress raw log files that are already generated (which requires CLP as a standalone tool)?

A bit more background:

CLP started as a standalone tool that takes raw log files as input, compresses them into a compressed format that we call archive, and then the archives can be searched without full decompression. The compression ratio of CLP archives is about 2x compared to directly compressing logs using Gzip or Zstandard.

But one issue with CLP is that compression requires hundreds of MBs of memory (in order for it to buffer enough parsed logs before compressing them into a columnar format). This overhead may not be acceptable for resource-constrained environments, so we developed a new real-time, resource-efficient compression. It uses a streaming compression protocol to compress the logs into an intermediate format (IR) without any buffering—this streaming compression is what we implemented in the logging libraries mentioned above. The IR can later be aggregated into archives by CLP. The architecture is shown below.

The compressed IR (IR for short) has the following advantages over directly compressing logs using general-purpose compressors:

• IR achieves moderate better compression compared to general purpose compressors (15% higher).

• More importantly, the logs are parsed before being compressed into IR—it recognizes different fields like timestamp, verbosity, static text and variables. Logs in the IR form can be viewed by our log viewer, which provides log-specific support such as filtering by timestamp, verbosity, and hiding certain log types (static text).

• We provide tools to "grep" through the IR for search.

• Compressing IR into CLP archives is much faster than compressing raw log files, because the logs are already parsed.

So if we built an Android logging library that outputs compressed IR, you could use it to compress logs directly from your apps. Then, you could view those compressed logs in our log viewer, further compress them using CLP, and also search them using CLP.

[TimLoong]

Yeap，I am trying to compress logs on AOSP as a test.
If possible, I hope that the CPL library can be migrated to more Android based embedded devices.

[bdchello]

hi diy，
Can you tell me your contact information? I would like to chat with you

[diy1]

Please contact us at [email protected].