@Override public void serialize( Text text, JsonGenerator jsonGenerator, SerializerProvider serializerProvider) throws IOException, JsonGenerationException { jsonGenerator.writeString(text.toString()); } }
public VarChar(String obj) { super(obj == null ? null : new Text(obj)); }
public static boolean castToInteger(FieldReader reader, NullableBigIntHolder out) { Object o = reader.readObject(); if (o instanceof Number) { out.value = ((Number) o).longValue(); return true; } else if (o instanceof Boolean) { out.value = ((Boolean) o).booleanValue() ? 1 : 0; return true; } else if (o instanceof LocalDateTime) { out.value = toMillis((LocalDateTime) o); return true; } else if (o instanceof Text) { try { String s = Text.decode(((Text) o).getBytes(), 0, ((Text) o).getLength()); return parseLong(s, out); } catch (CharacterCodingException e) { // TODO: is this the best way? logger.warn("Can't decode text", e); return false; } } else if (o instanceof byte[]) { return false; // TODO } return false; }
@Override public String toString() { try { return decode(bytes, 0, length); } catch (CharacterCodingException e) { throw new RuntimeException("Should not have happened ", e); } }
@Override public boolean equals(Object o) { if (o == this) { return true; } else if (o == null) { return false; } if (!(o instanceof Text)) { return false; } final Text that = (Text) o; if (this.getLength() != that.getLength()) { return false; } // copied from Arrays.equals so we don'thave to copy the byte arrays for (int i = 0; i < length; i++) { if (bytes[i] != that.bytes[i]) { return false; } } return true; }
/** * Returns the Unicode Scalar Value (32-bit integer value) for the character at * <code>position</code>. Note that this method avoids using the converter or doing String * instantiation. * * @param position the index of the char we want to retrieve * @return the Unicode scalar value at position or -1 if the position is invalid or points to a * trailing byte */ public int charAt(int position) { if (position > this.length) { return -1; // too long } if (position < 0) { return -1; // duh. } ByteBuffer bb = (ByteBuffer) ByteBuffer.wrap(bytes).position(position); return bytesToCodePoint(bb.slice()); }
/** * Get the variable length element at specified index as Text. * * @param index position of element to get * @return Text object for non-null element, null otherwise */ public Text getObject(int index) { Text result = new Text(); byte[] b; try { b = get(index); } catch (IllegalStateException e) { return null; } result.set(b); return result; }
/** * Converts the provided byte array to a String using the UTF-8 encoding. If the input is * malformed, replace by a default value. * * @param utf8 bytes to decode * @return the decoded string * @throws CharacterCodingException if this is not valid UTF-8 */ public static String decode(byte[] utf8) throws CharacterCodingException { return decode(ByteBuffer.wrap(utf8), true); }
public static boolean castToFloat(FieldReader reader, NullableFloat8Holder out) { Object o = reader.readObject(); if (o instanceof Number) { out.value = ((Number) o).doubleValue(); return true; } else if (o instanceof Boolean) { out.value = ((Boolean) o).booleanValue() ? 1 : 0; return true; } else if (o instanceof LocalDateTime) { out.value = toMillis((LocalDateTime) o); return true; } else if (o instanceof Text) { String s; try { s = Text.decode(((Text) o).getBytes(), 0, ((Text) o).getLength()); if (!isNumeric(s)) { return false; } out.value = Double.parseDouble(s); return true; } catch (Exception e) { // TODO: is this the best way? logger.warn("Can't decode text to FLOAT", e); return false; } } else if (o instanceof byte[]) { return false; // TODO } return false; }
public static List<String> getRefreshPath(final JobId jobId, final JobData jobData, final Path accelerationBasePath) { // extract written path from writer's metadata JobDataFragment data = jobData.range(0, 1); Text text = (Text) Preconditions.checkNotNull(data.extractValue(RecordWriter.PATH_COLUMN, 0), "Empty write path for job %s", jobId.getId()); // relative path to the acceleration base path final String path = PathUtils.relativePath(new Path(text.toString()), accelerationBasePath); // extract first 2 components of the path "<reflection-id>."<modified-materialization-id>" List<String> components = PathUtils.toPathComponents(path); Preconditions.checkState(components.size() >= 2, "Refresh path %s is incomplete", path); return ImmutableList.of(ACCELERATOR_STORAGEPLUGIN_NAME, components.get(0), components.get(1)); }
/** * Same as {@link #set(int, NullableVarCharHolder)} except that it handles the * case where index and length of new element are beyond the existing * capacity of the vector. * * @param index position of the element to set. * @param text Text object with data */ public void setSafe(int index, Text text) { setSafe(index, text.getBytes(), 0, text.getLength()); }
@Test public void testFileWithNulls() throws Exception { HashMap<String, FieldInfo> fields3 = new HashMap<>(); ParquetTestProperties props3 = new ParquetTestProperties(1, 3000, DEFAULT_BYTES_PER_PAGE, fields3); // actually include null values Object[] valuesWithNull = {new Text(""), new Text("longer string"), null}; props3.fields.put("a", new FieldInfo("boolean", "a", 1, valuesWithNull, TypeProtos.MinorType.BIT, props3)); testParquetFullEngineEventBased(false, "/parquet/parquet_scan_screen_read_entry_replace.json", "\"/tmp/nullable_with_nulls.parquet\"", "unused", 1, props3); }
public static String decode(byte[] utf8, int start, int length) throws CharacterCodingException { return decode(ByteBuffer.wrap(utf8, start, length), true); }
@Override public void writeVarChar(FieldReader reader) throws IOException { if (reader.isSet()) { writeVarChar(reader.readText().toString()); } else { writeVarcharNull(); } }